2024-09-18T14:08:48.089 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-18T14:08:48.102 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T14:08:48.176 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911098 branch: wip-bharath11-testing-2024-09-18-1115-quincy description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} email: skanta@redhat.com first_in_suite: false job_id: '7911098' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-bharath11-testing-2024-09-18-1115-quincy ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 workunit: branch: wip-bharath11-testing-2024-09-18-1115-quincy sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 4631 sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath11-testing-2024-09-18-1115-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 targets: smithi046.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-18_09:13:00 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-18T14:08:48.176 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa; will attempt to use it 2024-09-18T14:08:48.177 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa/tasks 2024-09-18T14:08:48.177 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-18T14:08:48.178 INFO:teuthology.task.internal:Checking packages... 2024-09-18T14:08:48.201 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '7061ea6ca50908d104e9f3979dc615e118d9e153' 2024-09-18T14:08:48.201 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-18T14:08:48.201 INFO:teuthology.packaging:ref: None 2024-09-18T14:08:48.201 INFO:teuthology.packaging:tag: None 2024-09-18T14:08:48.201 INFO:teuthology.packaging:branch: wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T14:08:48.201 INFO:teuthology.packaging:sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T14:08:48.201 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T14:08:48.345 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1645-g7061ea6c-1focal 2024-09-18T14:08:48.346 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-18T14:08:48.353 INFO:teuthology.task.internal:no buildpackages task found 2024-09-18T14:08:48.353 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-18T14:08:48.392 INFO:teuthology.task.internal:Saving configuration 2024-09-18T14:08:48.400 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-18T14:08:48.426 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-18T14:08:48.444 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911098', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-18 14:02:44.191335', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-18T14:08:48.445 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-18T14:08:48.458 INFO:teuthology.task.internal:roles: ubuntu@smithi046.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-18T14:08:48.459 INFO:teuthology.run_tasks:Running task console_log... 2024-09-18T14:08:48.521 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f605bcafac0>, signals=[15]) 2024-09-18T14:08:48.522 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-18T14:08:48.528 INFO:teuthology.task.internal:Opening connections... 2024-09-18T14:08:48.529 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi046.front.sepia.ceph.com 2024-09-18T14:08:48.531 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T14:08:48.621 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-18T14:08:48.627 DEBUG:teuthology.orchestra.run.smithi046:> uname -m 2024-09-18T14:08:48.747 INFO:teuthology.orchestra.run.smithi046.stdout:x86_64 2024-09-18T14:08:48.748 DEBUG:teuthology.orchestra.run.smithi046:> cat /etc/os-release 2024-09-18T14:08:48.754 INFO:teuthology.orchestra.run.smithi046.stdout:NAME="Ubuntu" 2024-09-18T14:08:48.754 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-18T14:08:48.754 INFO:teuthology.orchestra.run.smithi046.stdout:ID=ubuntu 2024-09-18T14:08:48.754 INFO:teuthology.orchestra.run.smithi046.stdout:ID_LIKE=debian 2024-09-18T14:08:48.755 INFO:teuthology.orchestra.run.smithi046.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-18T14:08:48.755 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_ID="20.04" 2024-09-18T14:08:48.755 INFO:teuthology.orchestra.run.smithi046.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-18T14:08:48.755 INFO:teuthology.orchestra.run.smithi046.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-18T14:08:48.755 INFO:teuthology.orchestra.run.smithi046.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-18T14:08:48.755 INFO:teuthology.orchestra.run.smithi046.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-18T14:08:48.755 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_CODENAME=focal 2024-09-18T14:08:48.755 INFO:teuthology.orchestra.run.smithi046.stdout:UBUNTU_CODENAME=focal 2024-09-18T14:08:48.756 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2024-09-18T14:08:48.782 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-18T14:08:48.790 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-18T14:08:48.817 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-18T14:08:48.818 DEBUG:teuthology.orchestra.run.smithi046:> test '!' -e /home/ubuntu/cephtest 2024-09-18T14:08:48.823 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-18T14:08:48.850 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-18T14:08:48.850 DEBUG:teuthology.orchestra.run.smithi046:> test -z $(ls -A /var/lib/ceph) 2024-09-18T14:08:48.860 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-18T14:08:48.909 INFO:teuthology.run_tasks:Running task kernel... 2024-09-18T14:08:48.923 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-18T14:08:48.924 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-18T14:08:48.924 DEBUG:teuthology.orchestra.run.smithi046:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-18T14:08:48.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T14:08:48.930 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2024-09-18T14:08:48.936 INFO:teuthology.orchestra.run.smithi046.stdout:5.4.0-124-generic 2024-09-18T14:08:48.936 INFO:teuthology.task.kernel:Running kernel on smithi046: 5.4.0-124-generic 2024-09-18T14:08:48.936 DEBUG:teuthology.orchestra.run.smithi046:> sudo apt-get clean 2024-09-18T14:08:49.022 DEBUG:teuthology.orchestra.run.smithi046:> sudo apt-get update 2024-09-18T14:08:49.317 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T14:08:49.327 INFO:teuthology.orchestra.run.smithi046.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-18T14:08:49.375 INFO:teuthology.orchestra.run.smithi046.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-18T14:08:49.728 INFO:teuthology.orchestra.run.smithi046.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-18T14:08:49.833 INFO:teuthology.orchestra.run.smithi046.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [817 kB] 2024-09-18T14:08:49.975 INFO:teuthology.orchestra.run.smithi046.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,036 kB] 2024-09-18T14:08:50.183 INFO:teuthology.orchestra.run.smithi046.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,223 kB] 2024-09-18T14:08:50.304 INFO:teuthology.orchestra.run.smithi046.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,597 kB] 2024-09-18T14:08:50.375 INFO:teuthology.orchestra.run.smithi046.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [475 kB] 2024-09-18T14:08:50.389 INFO:teuthology.orchestra.run.smithi046.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-18T14:08:50.390 INFO:teuthology.orchestra.run.smithi046.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,152 kB] 2024-09-18T14:08:50.480 INFO:teuthology.orchestra.run.smithi046.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-18T14:08:50.485 INFO:teuthology.orchestra.run.smithi046.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [553 kB] 2024-09-18T14:08:50.485 INFO:teuthology.orchestra.run.smithi046.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [441 kB] 2024-09-18T14:08:50.495 INFO:teuthology.orchestra.run.smithi046.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-18T14:08:50.495 INFO:teuthology.orchestra.run.smithi046.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-18T14:08:50.498 INFO:teuthology.orchestra.run.smithi046.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-18T14:08:50.498 INFO:teuthology.orchestra.run.smithi046.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,279 kB] 2024-09-18T14:08:50.514 INFO:teuthology.orchestra.run.smithi046.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,012 kB] 2024-09-18T14:08:50.544 INFO:teuthology.orchestra.run.smithi046.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-18T14:08:50.551 INFO:teuthology.orchestra.run.smithi046.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.4 kB] 2024-09-18T14:08:50.552 INFO:teuthology.orchestra.run.smithi046.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-18T14:08:50.552 INFO:teuthology.orchestra.run.smithi046.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-18T14:08:50.591 INFO:teuthology.orchestra.run.smithi046.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-18T14:08:50.591 INFO:teuthology.orchestra.run.smithi046.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [459 kB] 2024-09-18T14:08:50.609 INFO:teuthology.orchestra.run.smithi046.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-18T14:08:50.609 INFO:teuthology.orchestra.run.smithi046.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-18T14:08:50.628 INFO:teuthology.orchestra.run.smithi046.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-18T14:08:50.629 INFO:teuthology.orchestra.run.smithi046.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-18T14:08:50.636 INFO:teuthology.orchestra.run.smithi046.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,231 kB] 2024-09-18T14:08:50.665 INFO:teuthology.orchestra.run.smithi046.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-18T14:08:50.672 INFO:teuthology.orchestra.run.smithi046.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.3 kB] 2024-09-18T14:08:50.673 INFO:teuthology.orchestra.run.smithi046.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-18T14:08:50.673 INFO:teuthology.orchestra.run.smithi046.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-18T14:08:50.734 INFO:teuthology.orchestra.run.smithi046.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-18T14:08:50.734 INFO:teuthology.orchestra.run.smithi046.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-18T14:08:50.734 INFO:teuthology.orchestra.run.smithi046.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-18T14:08:50.735 INFO:teuthology.orchestra.run.smithi046.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-18T14:08:50.735 INFO:teuthology.orchestra.run.smithi046.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-18T14:08:50.815 INFO:teuthology.orchestra.run.smithi046.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-18T14:08:50.816 INFO:teuthology.orchestra.run.smithi046.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-18T14:08:50.816 INFO:teuthology.orchestra.run.smithi046.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-18T14:08:54.854 INFO:teuthology.orchestra.run.smithi046.stdout:Fetched 22.0 MB in 4s (5,922 kB/s) 2024-09-18T14:08:56.456 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2024-09-18T14:08:56.485 DEBUG:teuthology.orchestra.run.smithi046:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-18T14:08:56.566 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2024-09-18T14:08:56.791 INFO:teuthology.orchestra.run.smithi046.stdout:Building dependency tree... 2024-09-18T14:08:56.792 INFO:teuthology.orchestra.run.smithi046.stdout:Reading state information... 2024-09-18T14:08:57.012 INFO:teuthology.orchestra.run.smithi046.stdout:The following additional packages will be installed: 2024-09-18T14:08:57.012 INFO:teuthology.orchestra.run.smithi046.stdout: linux-generic linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-18T14:08:57.012 INFO:teuthology.orchestra.run.smithi046.stdout: linux-headers-generic linux-image-5.4.0-196-generic 2024-09-18T14:08:57.013 INFO:teuthology.orchestra.run.smithi046.stdout: linux-modules-5.4.0-196-generic linux-modules-extra-5.4.0-196-generic 2024-09-18T14:08:57.014 INFO:teuthology.orchestra.run.smithi046.stdout:Suggested packages: 2024-09-18T14:08:57.014 INFO:teuthology.orchestra.run.smithi046.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-18T14:08:57.014 INFO:teuthology.orchestra.run.smithi046.stdout:Recommended packages: 2024-09-18T14:08:57.014 INFO:teuthology.orchestra.run.smithi046.stdout: thermald 2024-09-18T14:08:57.076 INFO:teuthology.orchestra.run.smithi046.stdout:The following NEW packages will be installed: 2024-09-18T14:08:57.076 INFO:teuthology.orchestra.run.smithi046.stdout: linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-18T14:08:57.076 INFO:teuthology.orchestra.run.smithi046.stdout: linux-image-5.4.0-196-generic linux-modules-5.4.0-196-generic 2024-09-18T14:08:57.077 INFO:teuthology.orchestra.run.smithi046.stdout: linux-modules-extra-5.4.0-196-generic 2024-09-18T14:08:57.078 INFO:teuthology.orchestra.run.smithi046.stdout:The following packages will be upgraded: 2024-09-18T14:08:57.078 INFO:teuthology.orchestra.run.smithi046.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-18T14:08:57.133 INFO:teuthology.orchestra.run.smithi046.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-18T14:08:57.133 INFO:teuthology.orchestra.run.smithi046.stdout:Need to get 77.2 MB of archives. 2024-09-18T14:08:57.133 INFO:teuthology.orchestra.run.smithi046.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-18T14:08:57.133 INFO:teuthology.orchestra.run.smithi046.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-196-generic amd64 5.4.0-196.216 [15.0 MB] 2024-09-18T14:08:57.454 INFO:teuthology.orchestra.run.smithi046.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-196-generic amd64 5.4.0-196.216 [10.5 MB] 2024-09-18T14:08:57.549 INFO:teuthology.orchestra.run.smithi046.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-196-generic amd64 5.4.0-196.216 [39.2 MB] 2024-09-18T14:08:58.031 INFO:teuthology.orchestra.run.smithi046.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.196.194 [1,900 B] 2024-09-18T14:08:58.031 INFO:teuthology.orchestra.run.smithi046.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.196.194 [2,592 B] 2024-09-18T14:08:58.031 INFO:teuthology.orchestra.run.smithi046.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196 all 5.4.0-196.216 [11.0 MB] 2024-09-18T14:08:58.129 INFO:teuthology.orchestra.run.smithi046.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196-generic amd64 5.4.0-196.216 [1,386 kB] 2024-09-18T14:08:58.139 INFO:teuthology.orchestra.run.smithi046.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.196.194 [2,460 B] 2024-09-18T14:08:58.832 INFO:teuthology.orchestra.run.smithi046.stdout:Fetched 77.2 MB in 1s (73.2 MB/s) 2024-09-18T14:08:59.017 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package linux-modules-5.4.0-196-generic. 2024-09-18T14:09:00.692 INFO:teuthology.orchestra.run.smithi046.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-18T14:09:00.697 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../0-linux-modules-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T14:09:00.822 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:09:02.865 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package linux-image-5.4.0-196-generic. 2024-09-18T14:09:02.879 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../1-linux-image-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T14:09:03.015 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:09:04.135 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-196-generic. 2024-09-18T14:09:04.149 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T14:09:04.201 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:09:08.930 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../3-linux-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T14:09:09.071 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T14:09:09.433 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T14:09:09.566 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-image-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T14:09:09.894 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package linux-headers-5.4.0-196. 2024-09-18T14:09:09.911 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../5-linux-headers-5.4.0-196_5.4.0-196.216_all.deb ... 2024-09-18T14:09:10.244 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-18T14:09:15.813 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package linux-headers-5.4.0-196-generic. 2024-09-18T14:09:15.835 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../6-linux-headers-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T14:09:15.880 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:09:18.718 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T14:09:18.862 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-headers-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T14:09:19.196 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-18T14:09:19.334 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:09:19.484 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-headers-generic (5.4.0.196.194) ... 2024-09-18T14:09:19.635 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:09:20.887 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:09:22.062 INFO:teuthology.orchestra.run.smithi046.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-196-generic 2024-09-18T14:09:22.062 INFO:teuthology.orchestra.run.smithi046.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-196-generic 2024-09-18T14:09:22.198 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:09:23.416 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-image-generic (5.4.0.196.194) ... 2024-09-18T14:09:23.575 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-generic (5.4.0.196.194) ... 2024-09-18T14:09:23.718 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T14:09:23.794 INFO:teuthology.orchestra.run.smithi046.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-18T14:09:23.794 INFO:teuthology.orchestra.run.smithi046.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-196-generic 2024-09-18T14:09:46.785 INFO:teuthology.orchestra.run.smithi046.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-18T14:09:46.786 INFO:teuthology.orchestra.run.smithi046.stdout:Sourcing file `/etc/default/grub' 2024-09-18T14:09:46.794 INFO:teuthology.orchestra.run.smithi046.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-18T14:09:46.819 INFO:teuthology.orchestra.run.smithi046.stdout:Generating grub configuration file ... 2024-09-18T14:09:47.196 INFO:teuthology.orchestra.run.smithi046.stdout:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-18T14:09:47.223 INFO:teuthology.orchestra.run.smithi046.stdout:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-18T14:09:47.651 INFO:teuthology.orchestra.run.smithi046.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-18T14:09:47.663 INFO:teuthology.orchestra.run.smithi046.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-18T14:09:49.956 INFO:teuthology.orchestra.run.smithi046.stdout:done 2024-09-18T14:09:50.399 DEBUG:teuthology.orchestra.run.smithi046:> dpkg -s linux-image-generic 2024-09-18T14:09:50.444 INFO:teuthology.orchestra.run.smithi046.stdout:Package: linux-image-generic 2024-09-18T14:09:50.445 INFO:teuthology.orchestra.run.smithi046.stdout:Status: install ok installed 2024-09-18T14:09:50.445 INFO:teuthology.orchestra.run.smithi046.stdout:Priority: optional 2024-09-18T14:09:50.445 INFO:teuthology.orchestra.run.smithi046.stdout:Section: kernel 2024-09-18T14:09:50.445 INFO:teuthology.orchestra.run.smithi046.stdout:Installed-Size: 20 2024-09-18T14:09:50.445 INFO:teuthology.orchestra.run.smithi046.stdout:Maintainer: Ubuntu Kernel Team 2024-09-18T14:09:50.445 INFO:teuthology.orchestra.run.smithi046.stdout:Architecture: amd64 2024-09-18T14:09:50.445 INFO:teuthology.orchestra.run.smithi046.stdout:Source: linux-meta 2024-09-18T14:09:50.445 INFO:teuthology.orchestra.run.smithi046.stdout:Version: 5.4.0.196.194 2024-09-18T14:09:50.445 INFO:teuthology.orchestra.run.smithi046.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-18T14:09:50.445 INFO:teuthology.orchestra.run.smithi046.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-18T14:09:50.445 INFO:teuthology.orchestra.run.smithi046.stdout:Recommends: thermald 2024-09-18T14:09:50.446 INFO:teuthology.orchestra.run.smithi046.stdout:Description: Generic Linux kernel image 2024-09-18T14:09:50.446 INFO:teuthology.orchestra.run.smithi046.stdout: This package will always depend on the latest generic kernel image 2024-09-18T14:09:50.446 INFO:teuthology.orchestra.run.smithi046.stdout: available. 2024-09-18T14:09:50.446 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-196-generic 2024-09-18T14:09:50.446 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-18T14:09:50.446 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-18T14:09:50.446 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-18T14:09:50.447 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi046.front.sepia.ceph.com, path=None, version=distro) 2024-09-18T14:09:50.447 DEBUG:teuthology.orchestra.run.smithi046:> sudo apt-get clean 2024-09-18T14:09:50.532 DEBUG:teuthology.orchestra.run.smithi046:> sudo apt-get update 2024-09-18T14:09:50.666 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-18T14:09:50.783 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T14:09:50.864 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-18T14:09:50.946 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-18T14:09:52.600 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2024-09-18T14:09:52.626 DEBUG:teuthology.orchestra.run.smithi046:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-18T14:09:52.695 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2024-09-18T14:09:52.920 INFO:teuthology.orchestra.run.smithi046.stdout:Building dependency tree... 2024-09-18T14:09:52.921 INFO:teuthology.orchestra.run.smithi046.stdout:Reading state information... 2024-09-18T14:09:53.167 INFO:teuthology.orchestra.run.smithi046.stdout:linux-image-generic is already the newest version (5.4.0.196.194). 2024-09-18T14:09:53.168 INFO:teuthology.orchestra.run.smithi046.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-18T14:09:53.169 DEBUG:teuthology.orchestra.run.smithi046:> dpkg -s linux-image-generic 2024-09-18T14:09:53.192 INFO:teuthology.orchestra.run.smithi046.stdout:Package: linux-image-generic 2024-09-18T14:09:53.193 INFO:teuthology.orchestra.run.smithi046.stdout:Status: install ok installed 2024-09-18T14:09:53.193 INFO:teuthology.orchestra.run.smithi046.stdout:Priority: optional 2024-09-18T14:09:53.193 INFO:teuthology.orchestra.run.smithi046.stdout:Section: kernel 2024-09-18T14:09:53.193 INFO:teuthology.orchestra.run.smithi046.stdout:Installed-Size: 20 2024-09-18T14:09:53.193 INFO:teuthology.orchestra.run.smithi046.stdout:Maintainer: Ubuntu Kernel Team 2024-09-18T14:09:53.193 INFO:teuthology.orchestra.run.smithi046.stdout:Architecture: amd64 2024-09-18T14:09:53.193 INFO:teuthology.orchestra.run.smithi046.stdout:Source: linux-meta 2024-09-18T14:09:53.193 INFO:teuthology.orchestra.run.smithi046.stdout:Version: 5.4.0.196.194 2024-09-18T14:09:53.193 INFO:teuthology.orchestra.run.smithi046.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-18T14:09:53.193 INFO:teuthology.orchestra.run.smithi046.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-18T14:09:53.193 INFO:teuthology.orchestra.run.smithi046.stdout:Recommends: thermald 2024-09-18T14:09:53.193 INFO:teuthology.orchestra.run.smithi046.stdout:Description: Generic Linux kernel image 2024-09-18T14:09:53.194 INFO:teuthology.orchestra.run.smithi046.stdout: This package will always depend on the latest generic kernel image 2024-09-18T14:09:53.194 INFO:teuthology.orchestra.run.smithi046.stdout: available. 2024-09-18T14:09:53.194 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2024-09-18T14:09:53.198 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.k5evpTtDnb 2024-09-18T14:09:53.199 DEBUG:teuthology.orchestra.run.smithi046:> sudo cp /boot/grub/grub.cfg /tmp/tmp.k5evpTtDnb 2024-09-18T14:09:53.255 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.k5evpTtDnb 2024-09-18T14:09:53.301 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.k5evpTtDnb is 9KB 2024-09-18T14:09:53.316 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.k5evpTtDnb 2024-09-18T14:09:53.322 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-18T14:09:53.378 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-18T14:09:53.378 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-18T14:09:53.378 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-18T14:09:53.412 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-196-generic 2024-09-18T14:09:53.413 DEBUG:teuthology.orchestra.run.smithi046:> sudo update-grub 2024-09-18T14:09:54.859 INFO:teuthology.orchestra.run.smithi046.stderr:Sourcing file `/etc/default/grub' 2024-09-18T14:09:54.865 INFO:teuthology.orchestra.run.smithi046.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-18T14:09:54.876 INFO:teuthology.orchestra.run.smithi046.stderr:Generating grub configuration file ... 2024-09-18T14:09:55.138 INFO:teuthology.orchestra.run.smithi046.stderr:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-18T14:09:55.149 INFO:teuthology.orchestra.run.smithi046.stderr:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-18T14:09:55.549 INFO:teuthology.orchestra.run.smithi046.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-18T14:09:55.561 INFO:teuthology.orchestra.run.smithi046.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-18T14:09:56.622 INFO:teuthology.orchestra.run.smithi046.stderr:done 2024-09-18T14:09:56.624 DEBUG:teuthology.orchestra.run.smithi046:> sudo shutdown -r now 2024-09-18T14:10:26.657 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-18T14:10:26.658 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-09-18T14:10:26.659 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T14:11:01.309 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2024-09-18T14:11:10.318 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-09-18T14:11:10.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T14:12:10.382 DEBUG:teuthology.orchestra.remote:timed out 2024-09-18T14:12:22.394 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-09-18T14:12:22.395 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T14:12:22.649 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-09-18T14:12:23.263 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-09-18T14:12:23.263 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-196-generic"... 2024-09-18T14:12:23.263 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2024-09-18T14:12:23.270 INFO:teuthology.orchestra.run.smithi046.stdout:5.4.0-196-generic 2024-09-18T14:12:23.270 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-196-generic vs 5.4.0-196-generic 2024-09-18T14:12:23.270 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-18T14:12:23.270 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-18T14:12:24.272 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-18T14:12:24.272 DEBUG:teuthology.orchestra.run.smithi046:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-18T14:12:24.369 INFO:teuthology.orchestra.run.smithi046.stdout:ttyS1 2024-09-18T14:12:24.394 DEBUG:teuthology.parallel:result is None 2024-09-18T14:12:24.394 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-18T14:12:24.401 INFO:teuthology.task.internal:Creating test directory... 2024-09-18T14:12:24.401 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-18T14:12:24.409 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-18T14:12:24.445 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-18T14:12:24.478 INFO:teuthology.task.internal:Creating archive directory... 2024-09-18T14:12:24.478 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-18T14:12:24.496 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-18T14:12:24.515 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-18T14:12:24.515 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-18T14:12:24.533 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:12:24.544 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:12:24.546 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-18T14:12:24.553 INFO:teuthology.task.internal:Configuring sudo... 2024-09-18T14:12:24.553 DEBUG:teuthology.orchestra.run.smithi046:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-18T14:12:24.602 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-18T14:12:24.611 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-18T14:12:24.612 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-18T14:12:24.618 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T14:12:24.624 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T14:12:24.674 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-18T14:12:24.674 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-18T14:12:24.693 DEBUG:teuthology.orchestra.run.smithi046:> sudo service rsyslog restart 2024-09-18T14:12:24.744 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-18T14:12:24.751 INFO:teuthology.task.internal:Starting timer... 2024-09-18T14:12:24.751 INFO:teuthology.run_tasks:Running task pcp... 2024-09-18T14:12:24.774 INFO:teuthology.run_tasks:Running task selinux... 2024-09-18T14:12:24.807 DEBUG:teuthology.task.selinux:Excluding smithi046: OS 'ubuntu' does not support SELinux 2024-09-18T14:12:24.807 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-18T14:12:24.807 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-18T14:12:24.807 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-18T14:12:24.808 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-18T14:12:24.840 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-18T14:12:24.853 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-18T14:12:24.854 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi046.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-18T14:18:00.386 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi046.front.sepia.ceph.com')] 2024-09-18T14:18:00.388 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-09-18T14:18:00.389 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T14:18:00.480 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-09-18T14:18:00.602 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-09-18T14:18:00.602 INFO:teuthology.run_tasks:Running task clock... 2024-09-18T14:18:00.613 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-18T14:18:00.613 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-18T14:18:00.614 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T14:18:00.649 INFO:teuthology.orchestra.run.smithi046.stdout:18 Sep 14:18:00 ntpd[13696]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-18T14:18:00.649 INFO:teuthology.orchestra.run.smithi046.stdout:18 Sep 14:18:00 ntpd[13696]: Command line: ntpd -gq 2024-09-18T14:18:00.650 INFO:teuthology.orchestra.run.smithi046.stdout:18 Sep 14:18:00 ntpd[13696]: proto: precision = 0.098 usec (-23) 2024-09-18T14:18:00.651 INFO:teuthology.orchestra.run.smithi046.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-18T14:18:00.651 INFO:teuthology.orchestra.run.smithi046.stdout:18 Sep 14:18:00 ntpd[13696]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-18T14:18:00.651 INFO:teuthology.orchestra.run.smithi046.stdout:18 Sep 14:18:00 ntpd[13696]: restrict ::: KOD does nothing without LIMITED. 2024-09-18T14:18:00.651 INFO:teuthology.orchestra.run.smithi046.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-18T14:18:00.652 INFO:teuthology.orchestra.run.smithi046.stdout:18 Sep 14:18:00 ntpd[13696]: Listen and drop on 0 v6wildcard [::]:123 2024-09-18T14:18:00.652 INFO:teuthology.orchestra.run.smithi046.stdout:18 Sep 14:18:00 ntpd[13696]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-18T14:18:00.652 INFO:teuthology.orchestra.run.smithi046.stdout:18 Sep 14:18:00 ntpd[13696]: Listen normally on 2 lo 127.0.0.1:123 2024-09-18T14:18:00.652 INFO:teuthology.orchestra.run.smithi046.stdout:18 Sep 14:18:00 ntpd[13696]: Listen normally on 3 ens1f0 172.21.15.46:123 2024-09-18T14:18:00.652 INFO:teuthology.orchestra.run.smithi046.stdout:18 Sep 14:18:00 ntpd[13696]: Listen normally on 4 lo [::1]:123 2024-09-18T14:18:00.652 INFO:teuthology.orchestra.run.smithi046.stdout:18 Sep 14:18:00 ntpd[13696]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15f8%3]:123 2024-09-18T14:18:00.652 INFO:teuthology.orchestra.run.smithi046.stdout:18 Sep 14:18:00 ntpd[13696]: Listening on routing socket on fd #22 for interface updates 2024-09-18T14:18:01.650 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:01 ntpd[13696]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:18:01.650 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:01 ntpd[13696]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:18:01.651 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:01 ntpd[13696]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:18:01.651 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:01 ntpd[13696]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:18:01.651 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:01 ntpd[13696]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:18:01.651 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:01 ntpd[13696]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:18:03.650 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:03 ntpd[13696]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:18:03.651 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:03 ntpd[13696]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:18:03.651 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:03 ntpd[13696]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:18:03.651 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:03 ntpd[13696]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:18:03.651 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:03 ntpd[13696]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:18:03.651 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:03 ntpd[13696]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:18:04.652 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:04 ntpd[13696]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:18:04.652 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:04 ntpd[13696]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:18:05.651 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:05 ntpd[13696]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:18:05.652 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:05 ntpd[13696]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:18:05.652 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:05 ntpd[13696]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:18:05.652 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:05 ntpd[13696]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:18:05.652 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:05 ntpd[13696]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:18:05.652 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:05 ntpd[13696]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:18:05.652 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:05 ntpd[13696]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:18:05.652 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:05 ntpd[13696]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:18:05.652 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:05 ntpd[13696]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:18:05.652 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:05 ntpd[13696]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:18:05.652 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:05 ntpd[13696]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:18:05.652 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:05 ntpd[13696]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:18:05.652 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:05 ntpd[13696]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:18:05.652 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:05 ntpd[13696]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:18:05.653 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:05 ntpd[13696]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:18:05.653 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:05 ntpd[13696]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:18:05.653 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:05 ntpd[13696]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:18:05.653 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:05 ntpd[13696]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:18:07.651 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:07 ntpd[13696]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:18:07.651 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:07 ntpd[13696]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:18:07.651 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:07 ntpd[13696]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:18:07.651 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:07 ntpd[13696]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:18:07.651 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:07 ntpd[13696]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:18:07.651 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:07 ntpd[13696]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:18:07.651 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:07 ntpd[13696]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:18:07.651 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:07 ntpd[13696]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:18:07.651 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:07 ntpd[13696]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:18:07.651 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:07 ntpd[13696]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:18:07.652 INFO:teuthology.orchestra.run.smithi046.stdout:18 Sep 14:18:07 ntpd[13696]: ntpd: time slew -0.001560 s 2024-09-18T14:18:07.652 INFO:teuthology.orchestra.run.smithi046.stdout:ntpd: time slew -0.001560s 2024-09-18T14:18:07.652 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:07 ntpd[13696]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-18T14:18:07.652 INFO:teuthology.orchestra.run.smithi046.stderr:18 Sep 14:18:07 ntpd[13696]: can't open /var/log/ntpstats/loopstats.20240918: Permission denied 2024-09-18T14:18:07.717 INFO:teuthology.orchestra.run.smithi046.stdout: remote refid st t when poll reach delay offset jitter 2024-09-18T14:18:07.717 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================== 2024-09-18T14:18:07.717 INFO:teuthology.orchestra.run.smithi046.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T14:18:07.717 INFO:teuthology.orchestra.run.smithi046.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T14:18:07.718 INFO:teuthology.orchestra.run.smithi046.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T14:18:07.718 INFO:teuthology.orchestra.run.smithi046.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T14:18:07.718 INFO:teuthology.run_tasks:Running task install... 2024-09-18T14:18:07.726 DEBUG:teuthology.task.install:project ceph 2024-09-18T14:18:07.727 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}} 2024-09-18T14:18:07.727 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'} 2024-09-18T14:18:07.727 INFO:teuthology.task.install:Using flavor: default 2024-09-18T14:18:07.732 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-18T14:18:07.732 INFO:teuthology.task.install:extra packages: [] 2024-09-18T14:18:07.733 DEBUG:teuthology.orchestra.run.smithi046:> sudo apt-key list | grep Ceph 2024-09-18T14:18:07.870 INFO:teuthology.orchestra.run.smithi046.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-18T14:18:07.932 INFO:teuthology.orchestra.run.smithi046.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-18T14:18:07.932 INFO:teuthology.orchestra.run.smithi046.stdout:uid [ unknown] Ceph.com (release key) 2024-09-18T14:18:07.933 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-18T14:18:07.933 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T14:18:08.076 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default/ 2024-09-18T14:18:08.076 INFO:teuthology.task.install.deb:Package version is 17.2.7-1645-g7061ea6c-1focal 2024-09-18T14:18:08.188 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-18T14:18:08.188 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-18T14:18:08.205 DEBUG:teuthology.orchestra.run.smithi046:> sudo apt-get update 2024-09-18T14:18:08.449 INFO:teuthology.orchestra.run.smithi046.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal InRelease 2024-09-18T14:18:08.479 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-18T14:18:08.479 INFO:teuthology.orchestra.run.smithi046.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-18T14:18:08.502 INFO:teuthology.orchestra.run.smithi046.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal Release.gpg 2024-09-18T14:18:08.514 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T14:18:08.524 INFO:teuthology.orchestra.run.smithi046.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main i386 Packages [3,390 B] 2024-09-18T14:18:08.557 INFO:teuthology.orchestra.run.smithi046.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-09-18T14:18:08.596 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-18T14:18:08.677 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-18T14:18:09.154 INFO:teuthology.orchestra.run.smithi046.stdout:Fetched 28.4 kB in 1s (39.0 kB/s) 2024-09-18T14:18:10.355 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2024-09-18T14:18:10.381 DEBUG:teuthology.orchestra.run.smithi046:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-1645-g7061ea6c-1focal cephadm=17.2.7-1645-g7061ea6c-1focal ceph-mds=17.2.7-1645-g7061ea6c-1focal ceph-mgr=17.2.7-1645-g7061ea6c-1focal ceph-common=17.2.7-1645-g7061ea6c-1focal ceph-fuse=17.2.7-1645-g7061ea6c-1focal ceph-test=17.2.7-1645-g7061ea6c-1focal ceph-volume=17.2.7-1645-g7061ea6c-1focal radosgw=17.2.7-1645-g7061ea6c-1focal python3-rados=17.2.7-1645-g7061ea6c-1focal python3-rgw=17.2.7-1645-g7061ea6c-1focal python3-cephfs=17.2.7-1645-g7061ea6c-1focal python3-rbd=17.2.7-1645-g7061ea6c-1focal libcephfs2=17.2.7-1645-g7061ea6c-1focal libcephfs-dev=17.2.7-1645-g7061ea6c-1focal librados2=17.2.7-1645-g7061ea6c-1focal librbd1=17.2.7-1645-g7061ea6c-1focal rbd-fuse=17.2.7-1645-g7061ea6c-1focal 2024-09-18T14:18:10.453 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2024-09-18T14:18:10.655 INFO:teuthology.orchestra.run.smithi046.stdout:Building dependency tree... 2024-09-18T14:18:10.656 INFO:teuthology.orchestra.run.smithi046.stdout:Reading state information... 2024-09-18T14:18:10.861 INFO:teuthology.orchestra.run.smithi046.stdout:The following packages were automatically installed and are no longer required: 2024-09-18T14:18:10.862 INFO:teuthology.orchestra.run.smithi046.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-18T14:18:10.862 INFO:teuthology.orchestra.run.smithi046.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-18T14:18:10.862 INFO:teuthology.orchestra.run.smithi046.stdout:The following additional packages will be installed: 2024-09-18T14:18:10.862 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-18T14:18:10.864 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-18T14:18:10.864 INFO:teuthology.orchestra.run.smithi046.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-18T14:18:10.864 INFO:teuthology.orchestra.run.smithi046.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-18T14:18:10.865 INFO:teuthology.orchestra.run.smithi046.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-18T14:18:10.865 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-18T14:18:10.865 INFO:teuthology.orchestra.run.smithi046.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-09-18T14:18:10.865 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-09-18T14:18:10.865 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-09-18T14:18:10.865 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-09-18T14:18:10.865 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-09-18T14:18:10.865 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-09-18T14:18:10.865 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-09-18T14:18:10.865 INFO:teuthology.orchestra.run.smithi046.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-18T14:18:10.865 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-18T14:18:10.865 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-09-18T14:18:10.865 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-09-18T14:18:10.865 INFO:teuthology.orchestra.run.smithi046.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-18T14:18:10.866 INFO:teuthology.orchestra.run.smithi046.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-09-18T14:18:10.866 INFO:teuthology.orchestra.run.smithi046.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-09-18T14:18:10.866 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-09-18T14:18:10.866 INFO:teuthology.orchestra.run.smithi046.stdout: socat xmlstarlet 2024-09-18T14:18:10.866 INFO:teuthology.orchestra.run.smithi046.stdout:Suggested packages: 2024-09-18T14:18:10.866 INFO:teuthology.orchestra.run.smithi046.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-18T14:18:10.866 INFO:teuthology.orchestra.run.smithi046.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-18T14:18:10.866 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-18T14:18:10.867 INFO:teuthology.orchestra.run.smithi046.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-18T14:18:10.867 INFO:teuthology.orchestra.run.smithi046.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-18T14:18:10.867 INFO:teuthology.orchestra.run.smithi046.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-18T14:18:10.867 INFO:teuthology.orchestra.run.smithi046.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-18T14:18:10.867 INFO:teuthology.orchestra.run.smithi046.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-18T14:18:10.867 INFO:teuthology.orchestra.run.smithi046.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-18T14:18:10.867 INFO:teuthology.orchestra.run.smithi046.stdout:Recommended packages: 2024-09-18T14:18:10.867 INFO:teuthology.orchestra.run.smithi046.stdout: btrfs-tools 2024-09-18T14:18:10.990 INFO:teuthology.orchestra.run.smithi046.stdout:The following NEW packages will be installed: 2024-09-18T14:18:10.991 INFO:teuthology.orchestra.run.smithi046.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-18T14:18:10.991 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-18T14:18:10.991 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-18T14:18:10.991 INFO:teuthology.orchestra.run.smithi046.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-18T14:18:10.991 INFO:teuthology.orchestra.run.smithi046.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-18T14:18:10.991 INFO:teuthology.orchestra.run.smithi046.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-18T14:18:10.991 INFO:teuthology.orchestra.run.smithi046.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-18T14:18:10.993 INFO:teuthology.orchestra.run.smithi046.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-09-18T14:18:10.993 INFO:teuthology.orchestra.run.smithi046.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-09-18T14:18:10.993 INFO:teuthology.orchestra.run.smithi046.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-18T14:18:10.993 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-09-18T14:18:10.993 INFO:teuthology.orchestra.run.smithi046.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-09-18T14:18:10.993 INFO:teuthology.orchestra.run.smithi046.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-09-18T14:18:10.993 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-09-18T14:18:10.993 INFO:teuthology.orchestra.run.smithi046.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-09-18T14:18:10.993 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-09-18T14:18:10.993 INFO:teuthology.orchestra.run.smithi046.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-18T14:18:10.994 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-18T14:18:10.994 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-09-18T14:18:10.994 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-09-18T14:18:10.994 INFO:teuthology.orchestra.run.smithi046.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-18T14:18:10.994 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-18T14:18:10.994 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-18T14:18:10.994 INFO:teuthology.orchestra.run.smithi046.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2024-09-18T14:18:10.994 INFO:teuthology.orchestra.run.smithi046.stdout:The following packages will be upgraded: 2024-09-18T14:18:10.994 INFO:teuthology.orchestra.run.smithi046.stdout: librados2 librbd1 2024-09-18T14:18:11.143 INFO:teuthology.orchestra.run.smithi046.stdout:2 upgraded, 108 newly installed, 0 to remove and 295 not upgraded. 2024-09-18T14:18:11.143 INFO:teuthology.orchestra.run.smithi046.stdout:Need to get 201 MB of archives. 2024-09-18T14:18:11.143 INFO:teuthology.orchestra.run.smithi046.stdout:After this operation, 938 MB of additional disk space will be used. 2024-09-18T14:18:11.143 INFO:teuthology.orchestra.run.smithi046.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-1645-g7061ea6c-1focal [3,507 kB] 2024-09-18T14:18:11.187 INFO:teuthology.orchestra.run.smithi046.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-18T14:18:11.332 INFO:teuthology.orchestra.run.smithi046.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-1645-g7061ea6c-1focal [3,562 kB] 2024-09-18T14:18:11.382 INFO:teuthology.orchestra.run.smithi046.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-1645-g7061ea6c-1focal [761 kB] 2024-09-18T14:18:11.394 INFO:teuthology.orchestra.run.smithi046.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-1645-g7061ea6c-1focal [414 kB] 2024-09-18T14:18:11.401 INFO:teuthology.orchestra.run.smithi046.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-1645-g7061ea6c-1focal [22.6 kB] 2024-09-18T14:18:11.401 INFO:teuthology.orchestra.run.smithi046.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-1645-g7061ea6c-1focal [200 kB] 2024-09-18T14:18:11.404 INFO:teuthology.orchestra.run.smithi046.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-1645-g7061ea6c-1focal [49.1 kB] 2024-09-18T14:18:11.405 INFO:teuthology.orchestra.run.smithi046.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-1645-g7061ea6c-1focal [390 kB] 2024-09-18T14:18:11.412 INFO:teuthology.orchestra.run.smithi046.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-1645-g7061ea6c-1focal [5,637 kB] 2024-09-18T14:18:11.505 INFO:teuthology.orchestra.run.smithi046.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-1645-g7061ea6c-1focal [116 kB] 2024-09-18T14:18:11.506 INFO:teuthology.orchestra.run.smithi046.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-1645-g7061ea6c-1focal [479 kB] 2024-09-18T14:18:11.514 INFO:teuthology.orchestra.run.smithi046.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-1645-g7061ea6c-1focal [27.0 MB] 2024-09-18T14:18:11.591 INFO:teuthology.orchestra.run.smithi046.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-18T14:18:11.598 INFO:teuthology.orchestra.run.smithi046.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-18T14:18:11.641 INFO:teuthology.orchestra.run.smithi046.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-18T14:18:11.655 INFO:teuthology.orchestra.run.smithi046.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-18T14:18:11.780 INFO:teuthology.orchestra.run.smithi046.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-18T14:18:11.786 INFO:teuthology.orchestra.run.smithi046.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-18T14:18:11.815 INFO:teuthology.orchestra.run.smithi046.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-18T14:18:11.956 INFO:teuthology.orchestra.run.smithi046.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-18T14:18:12.029 INFO:teuthology.orchestra.run.smithi046.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-18T14:18:12.092 INFO:teuthology.orchestra.run.smithi046.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-18T14:18:12.092 INFO:teuthology.orchestra.run.smithi046.stdout:Get:24 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-1645-g7061ea6c-1focal [5,177 kB] 2024-09-18T14:18:12.095 INFO:teuthology.orchestra.run.smithi046.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-18T14:18:12.096 INFO:teuthology.orchestra.run.smithi046.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-18T14:18:12.101 INFO:teuthology.orchestra.run.smithi046.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-18T14:18:12.102 INFO:teuthology.orchestra.run.smithi046.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-18T14:18:12.104 INFO:teuthology.orchestra.run.smithi046.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-18T14:18:12.104 INFO:teuthology.orchestra.run.smithi046.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-18T14:18:12.106 INFO:teuthology.orchestra.run.smithi046.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-18T14:18:12.119 INFO:teuthology.orchestra.run.smithi046.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-18T14:18:12.137 INFO:teuthology.orchestra.run.smithi046.stdout:Get:33 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-1645-g7061ea6c-1focal [223 kB] 2024-09-18T14:18:12.139 INFO:teuthology.orchestra.run.smithi046.stdout:Get:34 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-1645-g7061ea6c-1focal [157 kB] 2024-09-18T14:18:12.141 INFO:teuthology.orchestra.run.smithi046.stdout:Get:35 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-1645-g7061ea6c-1focal [1,594 kB] 2024-09-18T14:18:12.162 INFO:teuthology.orchestra.run.smithi046.stdout:Get:36 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-1645-g7061ea6c-1focal [4,688 kB] 2024-09-18T14:18:12.188 INFO:teuthology.orchestra.run.smithi046.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-18T14:18:12.189 INFO:teuthology.orchestra.run.smithi046.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-18T14:18:12.190 INFO:teuthology.orchestra.run.smithi046.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-18T14:18:12.192 INFO:teuthology.orchestra.run.smithi046.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-18T14:18:12.193 INFO:teuthology.orchestra.run.smithi046.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-18T14:18:12.193 INFO:teuthology.orchestra.run.smithi046.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-18T14:18:12.196 INFO:teuthology.orchestra.run.smithi046.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-18T14:18:12.197 INFO:teuthology.orchestra.run.smithi046.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-18T14:18:12.202 INFO:teuthology.orchestra.run.smithi046.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-18T14:18:12.240 INFO:teuthology.orchestra.run.smithi046.stdout:Get:46 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-1645-g7061ea6c-1focal [23.1 MB] 2024-09-18T14:18:12.283 INFO:teuthology.orchestra.run.smithi046.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-18T14:18:12.283 INFO:teuthology.orchestra.run.smithi046.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-18T14:18:12.292 INFO:teuthology.orchestra.run.smithi046.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-18T14:18:12.293 INFO:teuthology.orchestra.run.smithi046.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-18T14:18:12.294 INFO:teuthology.orchestra.run.smithi046.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-18T14:18:12.295 INFO:teuthology.orchestra.run.smithi046.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-18T14:18:12.297 INFO:teuthology.orchestra.run.smithi046.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-18T14:18:12.299 INFO:teuthology.orchestra.run.smithi046.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-18T14:18:12.365 INFO:teuthology.orchestra.run.smithi046.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-18T14:18:12.365 INFO:teuthology.orchestra.run.smithi046.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-18T14:18:12.446 INFO:teuthology.orchestra.run.smithi046.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-18T14:18:12.448 INFO:teuthology.orchestra.run.smithi046.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-18T14:18:12.450 INFO:teuthology.orchestra.run.smithi046.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-18T14:18:12.451 INFO:teuthology.orchestra.run.smithi046.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-18T14:18:12.452 INFO:teuthology.orchestra.run.smithi046.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-18T14:18:12.696 INFO:teuthology.orchestra.run.smithi046.stdout:Get:62 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-1645-g7061ea6c-1focal [3,936 B] 2024-09-18T14:18:12.696 INFO:teuthology.orchestra.run.smithi046.stdout:Get:63 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-1645-g7061ea6c-1focal [872 kB] 2024-09-18T14:18:12.707 INFO:teuthology.orchestra.run.smithi046.stdout:Get:64 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-1645-g7061ea6c-1focal [2,416 kB] 2024-09-18T14:18:12.725 INFO:teuthology.orchestra.run.smithi046.stdout:Get:65 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-1645-g7061ea6c-1focal [81.4 kB] 2024-09-18T14:18:12.725 INFO:teuthology.orchestra.run.smithi046.stdout:Get:66 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-1645-g7061ea6c-1focal [120 kB] 2024-09-18T14:18:12.727 INFO:teuthology.orchestra.run.smithi046.stdout:Get:67 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-1645-g7061ea6c-1focal [6,653 kB] 2024-09-18T14:18:12.776 INFO:teuthology.orchestra.run.smithi046.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-18T14:18:12.808 INFO:teuthology.orchestra.run.smithi046.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-18T14:18:12.811 INFO:teuthology.orchestra.run.smithi046.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-18T14:18:12.889 INFO:teuthology.orchestra.run.smithi046.stdout:Get:71 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-1645-g7061ea6c-1focal [7,884 kB] 2024-09-18T14:18:12.889 INFO:teuthology.orchestra.run.smithi046.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-18T14:18:12.889 INFO:teuthology.orchestra.run.smithi046.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-18T14:18:12.889 INFO:teuthology.orchestra.run.smithi046.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-18T14:18:12.889 INFO:teuthology.orchestra.run.smithi046.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-18T14:18:12.890 INFO:teuthology.orchestra.run.smithi046.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-18T14:18:12.890 INFO:teuthology.orchestra.run.smithi046.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-18T14:18:12.891 INFO:teuthology.orchestra.run.smithi046.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-18T14:18:12.940 INFO:teuthology.orchestra.run.smithi046.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-18T14:18:12.941 INFO:teuthology.orchestra.run.smithi046.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-18T14:18:12.975 INFO:teuthology.orchestra.run.smithi046.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-1645-g7061ea6c-1focal [4,056 B] 2024-09-18T14:18:12.975 INFO:teuthology.orchestra.run.smithi046.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-1645-g7061ea6c-1focal [53.4 MB] 2024-09-18T14:18:13.022 INFO:teuthology.orchestra.run.smithi046.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-18T14:18:13.027 INFO:teuthology.orchestra.run.smithi046.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-18T14:18:13.030 INFO:teuthology.orchestra.run.smithi046.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-18T14:18:13.033 INFO:teuthology.orchestra.run.smithi046.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-18T14:18:13.035 INFO:teuthology.orchestra.run.smithi046.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-18T14:18:13.039 INFO:teuthology.orchestra.run.smithi046.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-18T14:18:13.041 INFO:teuthology.orchestra.run.smithi046.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-18T14:18:13.111 INFO:teuthology.orchestra.run.smithi046.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-18T14:18:13.116 INFO:teuthology.orchestra.run.smithi046.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-18T14:18:13.283 INFO:teuthology.orchestra.run.smithi046.stdout:Get:92 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-18T14:18:13.284 INFO:teuthology.orchestra.run.smithi046.stdout:Get:93 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-18T14:18:13.284 INFO:teuthology.orchestra.run.smithi046.stdout:Get:94 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-18T14:18:13.285 INFO:teuthology.orchestra.run.smithi046.stdout:Get:95 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-18T14:18:13.285 INFO:teuthology.orchestra.run.smithi046.stdout:Get:96 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-18T14:18:13.286 INFO:teuthology.orchestra.run.smithi046.stdout:Get:97 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-18T14:18:13.299 INFO:teuthology.orchestra.run.smithi046.stdout:Get:98 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-18T14:18:13.299 INFO:teuthology.orchestra.run.smithi046.stdout:Get:99 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-18T14:18:13.300 INFO:teuthology.orchestra.run.smithi046.stdout:Get:100 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-18T14:18:13.367 INFO:teuthology.orchestra.run.smithi046.stdout:Get:101 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-18T14:18:13.448 INFO:teuthology.orchestra.run.smithi046.stdout:Get:102 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-18T14:18:13.449 INFO:teuthology.orchestra.run.smithi046.stdout:Get:103 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-18T14:18:13.449 INFO:teuthology.orchestra.run.smithi046.stdout:Get:104 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-18T14:18:13.453 INFO:teuthology.orchestra.run.smithi046.stdout:Get:105 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-18T14:18:13.454 INFO:teuthology.orchestra.run.smithi046.stdout:Get:106 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-18T14:18:14.068 INFO:teuthology.orchestra.run.smithi046.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-1645-g7061ea6c-1focal [231 kB] 2024-09-18T14:18:14.073 INFO:teuthology.orchestra.run.smithi046.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-1645-g7061ea6c-1focal [21.9 kB] 2024-09-18T14:18:14.075 INFO:teuthology.orchestra.run.smithi046.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-1645-g7061ea6c-1focal [17.2 MB] 2024-09-18T14:18:14.388 INFO:teuthology.orchestra.run.smithi046.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-1645-g7061ea6c-1focal [80.6 kB] 2024-09-18T14:18:15.055 INFO:teuthology.orchestra.run.smithi046.stdout:Fetched 201 MB in 3s (59.4 MB/s) 2024-09-18T14:18:15.242 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-18T14:18:15.300 INFO:teuthology.orchestra.run.smithi046.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-18T14:18:15.304 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-18T14:18:15.391 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-18T14:18:15.738 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../001-librbd1_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:18:15.861 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking librbd1 (17.2.7-1645-g7061ea6c-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-18T14:18:16.535 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../002-librados2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:18:16.682 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking librados2 (17.2.7-1645-g7061ea6c-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-18T14:18:17.386 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libcephfs2. 2024-09-18T14:18:17.406 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:18:17.445 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libcephfs2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:18:17.780 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-rados. 2024-09-18T14:18:17.797 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../004-python3-rados_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:18:17.838 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-rados (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:18:18.216 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-18T14:18:18.232 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:18:18.274 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-ceph-argparse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:18:18.510 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-cephfs. 2024-09-18T14:18:18.528 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:18:18.706 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-cephfs (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:18:19.038 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-18T14:18:19.055 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:18:19.096 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:18:19.440 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-prettytable. 2024-09-18T14:18:19.459 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-09-18T14:18:19.498 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-18T14:18:19.751 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-rbd. 2024-09-18T14:18:19.767 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:18:19.809 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-rbd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:18:20.145 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-18T14:18:20.161 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-18T14:18:20.203 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-18T14:18:20.548 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-18T14:18:20.565 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-18T14:18:20.606 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-18T14:18:20.967 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-18T14:18:20.984 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-18T14:18:21.025 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-18T14:18:21.429 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-18T14:18:21.445 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-18T14:18:21.487 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-18T14:18:21.815 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-18T14:18:21.832 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-18T14:18:21.873 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-18T14:18:22.368 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-18T14:18:22.384 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T14:18:22.436 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T14:18:22.971 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-18T14:18:22.988 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T14:18:23.021 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T14:18:23.374 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-18T14:18:23.390 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T14:18:23.453 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T14:18:23.835 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-18T14:18:23.852 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-18T14:18:23.893 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-18T14:18:24.255 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package librgw2. 2024-09-18T14:18:24.274 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../019-librgw2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:18:24.313 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking librgw2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:18:25.090 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-rgw. 2024-09-18T14:18:25.106 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:18:25.148 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-rgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:18:25.529 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package liboath0. 2024-09-18T14:18:25.547 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-18T14:18:25.593 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-18T14:18:25.895 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libradosstriper1. 2024-09-18T14:18:25.915 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:18:25.953 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libradosstriper1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:18:26.424 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-common. 2024-09-18T14:18:26.440 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../023-ceph-common_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:18:26.473 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:18:29.271 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-base. 2024-09-18T14:18:29.290 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../024-ceph-base_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:18:29.330 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-base (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:18:30.225 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-cryptography. 2024-09-18T14:18:30.241 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-18T14:18:30.283 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-18T14:18:30.678 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-openssl. 2024-09-18T14:18:30.698 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-18T14:18:30.736 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-18T14:18:31.089 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-packaging. 2024-09-18T14:18:31.107 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../027-python3-packaging_20.3-1_all.deb ... 2024-09-18T14:18:31.149 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-18T14:18:31.400 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-18T14:18:31.417 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../028-ceph-mgr-modules-core_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:18:31.458 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr-modules-core (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:18:31.846 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-18T14:18:31.864 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../029-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-18T14:18:31.904 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-18T14:18:32.207 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-18T14:18:32.224 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../030-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-18T14:18:32.265 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-18T14:18:32.635 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-natsort. 2024-09-18T14:18:32.651 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../031-python3-natsort_7.0.1-1_all.deb ... 2024-09-18T14:18:32.693 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-18T14:18:33.007 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-jwt. 2024-09-18T14:18:33.026 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../032-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-18T14:18:33.086 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-18T14:18:33.401 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-logutils. 2024-09-18T14:18:33.417 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../033-python3-logutils_0.3.3-7_all.deb ... 2024-09-18T14:18:33.459 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-18T14:18:33.787 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-18T14:18:33.804 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../034-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-18T14:18:33.845 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-18T14:18:34.157 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-mako. 2024-09-18T14:18:34.175 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../035-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-18T14:18:34.215 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-18T14:18:34.560 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-18T14:18:34.577 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../036-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-18T14:18:34.618 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-18T14:18:34.921 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-18T14:18:34.937 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../037-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-18T14:18:34.979 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-18T14:18:35.307 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-webob. 2024-09-18T14:18:35.327 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../038-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-18T14:18:35.365 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-18T14:18:35.699 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-18T14:18:35.717 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../039-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-18T14:18:35.757 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-18T14:18:36.094 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-bs4. 2024-09-18T14:18:36.110 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../040-python3-bs4_4.8.2-1_all.deb ... 2024-09-18T14:18:36.152 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-18T14:18:36.556 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-waitress. 2024-09-18T14:18:36.575 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../041-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-18T14:18:36.617 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-18T14:18:36.950 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-tempita. 2024-09-18T14:18:36.967 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6_all.deb ... 2024-09-18T14:18:37.008 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-18T14:18:37.328 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-paste. 2024-09-18T14:18:37.347 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../043-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-18T14:18:37.386 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-18T14:18:37.731 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-18T14:18:37.748 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-18T14:18:37.789 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-18T14:18:38.118 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-18T14:18:38.134 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-18T14:18:38.168 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-18T14:18:38.487 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-webtest. 2024-09-18T14:18:38.504 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../046-python3-webtest_2.0.32-2_all.deb ... 2024-09-18T14:18:38.587 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-18T14:18:38.901 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pecan. 2024-09-18T14:18:38.917 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-18T14:18:38.960 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-18T14:18:39.349 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-18T14:18:39.365 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../048-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-18T14:18:39.399 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-18T14:18:39.793 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-18T14:18:39.809 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../049-libsqlite3-mod-ceph_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:18:39.851 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:18:40.221 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr. 2024-09-18T14:18:40.237 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../050-ceph-mgr_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:18:40.278 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:18:40.790 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mon. 2024-09-18T14:18:40.810 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../051-ceph-mon_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:18:40.840 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mon (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:18:41.687 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-osd. 2024-09-18T14:18:41.704 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../052-ceph-osd_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:18:41.744 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-osd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:18:43.823 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph. 2024-09-18T14:18:43.838 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../053-ceph_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:18:43.880 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:18:44.175 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-fuse. 2024-09-18T14:18:44.195 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../054-ceph-fuse_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:18:44.225 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:18:45.395 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mds. 2024-09-18T14:18:45.415 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../055-ceph-mds_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:18:45.470 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mds (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:18:46.024 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package cephadm. 2024-09-18T14:18:46.043 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../056-cephadm_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:18:46.081 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:18:46.409 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-jinja2. 2024-09-18T14:18:46.426 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../057-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-18T14:18:46.467 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-18T14:18:46.829 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-18T14:18:46.845 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../058-ceph-mgr-cephadm_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:18:46.887 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr-cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:18:47.223 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-18T14:18:47.241 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../059-python3-repoze.lru_0.7-2_all.deb ... 2024-09-18T14:18:47.281 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-18T14:18:47.576 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libjs-underscore. 2024-09-18T14:18:47.593 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../060-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-18T14:18:47.634 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-18T14:18:47.903 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-18T14:18:47.919 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../061-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-18T14:18:47.961 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-18T14:18:48.281 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-routes. 2024-09-18T14:18:48.297 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../062-python3-routes_2.4.1-2_all.deb ... 2024-09-18T14:18:48.339 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-18T14:18:48.718 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-18T14:18:48.735 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../063-ceph-mgr-dashboard_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:18:48.775 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr-dashboard (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:18:50.124 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-decorator. 2024-09-18T14:18:50.141 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../064-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-18T14:18:50.182 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-18T14:18:50.468 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-18T14:18:50.488 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../065-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-18T14:18:50.535 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-18T14:18:50.854 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-scipy. 2024-09-18T14:18:50.872 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../066-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-18T14:18:50.912 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-18T14:18:52.213 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-18T14:18:52.230 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../067-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-18T14:18:52.271 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-18T14:18:52.850 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-joblib. 2024-09-18T14:18:52.866 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../068-python3-joblib_0.14.0-3_all.deb ... 2024-09-18T14:18:52.908 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-18T14:18:53.270 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-sklearn. 2024-09-18T14:18:53.286 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../069-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-18T14:18:53.328 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-18T14:18:53.961 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-18T14:18:53.978 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../070-ceph-mgr-diskprediction-local_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:18:54.019 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:18:55.147 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-cachetools. 2024-09-18T14:18:55.165 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../071-python3-cachetools_4.0.0-1_all.deb ... 2024-09-18T14:18:55.205 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-18T14:18:55.542 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-18T14:18:55.560 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../072-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-18T14:18:55.600 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-18T14:18:55.920 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-18T14:18:55.939 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../073-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-18T14:18:55.977 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-18T14:18:56.281 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-rsa. 2024-09-18T14:18:56.300 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../074-python3-rsa_4.0-3_all.deb ... 2024-09-18T14:18:56.339 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-18T14:18:56.643 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-google-auth. 2024-09-18T14:18:56.660 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../075-python3-google-auth_1.5.1-2_all.deb ... 2024-09-18T14:18:56.701 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-18T14:18:57.054 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-blinker. 2024-09-18T14:18:57.071 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../076-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-18T14:18:57.120 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-18T14:18:57.423 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-18T14:18:57.441 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../077-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-18T14:18:57.481 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-18T14:18:57.819 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-18T14:18:57.839 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-18T14:18:57.877 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-18T14:18:58.222 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-websocket. 2024-09-18T14:18:58.240 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../079-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-18T14:18:58.279 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-18T14:18:58.574 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-18T14:18:58.592 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../080-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-18T14:18:58.632 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-18T14:18:59.277 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-18T14:18:59.294 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:18:59.335 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:18:59.696 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-18T14:18:59.713 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../082-libonig5_6.9.4-1_amd64.deb ... 2024-09-18T14:18:59.754 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-18T14:19:00.091 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-18T14:19:00.109 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../083-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-18T14:19:00.141 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-18T14:19:00.453 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package jq. 2024-09-18T14:19:00.469 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../084-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-18T14:19:00.511 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-18T14:19:01.110 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package socat. 2024-09-18T14:19:01.129 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../085-socat_1.7.3.3-2_amd64.deb ... 2024-09-18T14:19:01.172 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-18T14:19:01.463 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package xmlstarlet. 2024-09-18T14:19:01.482 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-18T14:19:01.512 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-18T14:19:01.885 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-test. 2024-09-18T14:19:01.902 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../087-ceph-test_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:01.943 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-test (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:06.549 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-volume. 2024-09-18T14:19:06.565 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../088-ceph-volume_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:19:06.598 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-volume (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:06.908 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-18T14:19:06.924 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../089-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-18T14:19:06.963 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-18T14:19:07.467 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-18T14:19:07.484 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../090-libcephfs-dev_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:07.525 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libcephfs-dev (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:07.920 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package nvme-cli. 2024-09-18T14:19:07.940 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../091-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-18T14:19:07.996 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-18T14:19:08.325 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-18T14:19:08.345 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../092-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-18T14:19:08.383 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-18T14:19:10.095 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-18T14:19:10.115 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../093-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-18T14:19:10.152 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-18T14:19:10.464 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-18T14:19:10.484 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../094-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-18T14:19:10.555 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-18T14:19:10.867 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-attr. 2024-09-18T14:19:10.884 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../095-python3-attr_19.3.0-2_all.deb ... 2024-09-18T14:19:10.925 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-18T14:19:11.311 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-webencodings. 2024-09-18T14:19:11.328 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../096-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-18T14:19:11.369 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-18T14:19:11.664 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-html5lib. 2024-09-18T14:19:11.684 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../097-python3-html5lib_1.0.1-2_all.deb ... 2024-09-18T14:19:11.722 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-18T14:19:12.083 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-18T14:19:12.100 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../098-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-18T14:19:12.141 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-18T14:19:12.620 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pastescript. 2024-09-18T14:19:12.640 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-09-18T14:19:12.677 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-18T14:19:13.015 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pluggy. 2024-09-18T14:19:13.033 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-09-18T14:19:13.072 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-18T14:19:13.434 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-psutil. 2024-09-18T14:19:13.451 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-18T14:19:13.500 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-18T14:19:13.845 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-py. 2024-09-18T14:19:13.865 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-18T14:19:13.903 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-18T14:19:14.240 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-18T14:19:14.260 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-18T14:19:14.298 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-18T14:19:14.634 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-18T14:19:14.653 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-18T14:19:14.692 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-18T14:19:14.987 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pytest. 2024-09-18T14:19:15.008 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-09-18T14:19:15.045 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-18T14:19:15.427 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-simplejson. 2024-09-18T14:19:15.443 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-18T14:19:15.484 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-18T14:19:15.737 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-18T14:19:15.754 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-18T14:19:15.795 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-18T14:19:16.542 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package radosgw. 2024-09-18T14:19:16.559 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../108-radosgw_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:16.599 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking radosgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:18.378 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package rbd-fuse. 2024-09-18T14:19:18.397 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:19:18.436 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking rbd-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:18.738 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-18T14:19:19.068 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-18T14:19:19.194 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-18T14:19:19.295 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-18T14:19:19.485 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:20.033 INFO:teuthology.orchestra.run.smithi046.stdout:Adding system user cephadm....done 2024-09-18T14:19:20.211 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-18T14:19:20.312 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-18T14:19:20.649 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-18T14:19:20.941 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-18T14:19:21.067 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-18T14:19:21.193 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-18T14:19:21.537 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-18T14:19:21.922 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-18T14:19:22.215 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-18T14:19:22.491 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-18T14:19:22.775 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-18T14:19:23.093 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-ceph-argparse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:23.394 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-18T14:19:23.537 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-18T14:19:23.788 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-18T14:19:24.048 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-18T14:19:24.207 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-18T14:19:24.583 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-18T14:19:24.909 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-18T14:19:25.194 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-18T14:19:25.562 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-18T14:19:25.821 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-18T14:19:26.148 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-18T14:19:26.474 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-18T14:19:26.775 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-18T14:19:26.934 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-18T14:19:27.060 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-18T14:19:27.353 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-18T14:19:27.755 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-18T14:19:28.106 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-18T14:19:28.467 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-18T14:19:28.592 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:28.944 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T14:19:29.078 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-18T14:19:29.421 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-18T14:19:29.531 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-18T14:19:29.665 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-18T14:19:29.950 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-18T14:19:30.301 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T14:19:30.427 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-18T14:19:30.553 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-18T14:19:30.871 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-18T14:19:31.139 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-18T14:19:31.399 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-18T14:19:31.525 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-18T14:19:32.159 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-18T14:19:32.452 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-18T14:19:32.762 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-18T14:19:33.110 INFO:teuthology.orchestra.run.smithi046.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-18T14:19:33.381 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-18T14:19:33.499 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-18T14:19:33.817 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-18T14:19:34.177 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-18T14:19:34.302 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-18T14:19:34.454 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-18T14:19:34.780 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-18T14:19:35.048 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up librados2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:35.157 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-18T14:19:36.688 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libsqlite3-mod-ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:36.789 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-18T14:19:36.898 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-18T14:19:37.275 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-18T14:19:37.559 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-18T14:19:37.844 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T14:19:37.970 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-18T14:19:38.305 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libcephfs2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:38.447 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-18T14:19:38.758 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libradosstriper1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:38.884 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-18T14:19:39.210 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-18T14:19:40.274 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up librbd1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:40.425 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-18T14:19:40.551 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-18T14:19:40.677 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-18T14:19:40.803 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-18T14:19:41.104 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-18T14:19:41.222 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-18T14:19:41.508 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr-modules-core (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:41.634 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:41.880 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-18T14:19:41.880 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-18T14:19:42.336 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libcephfs-dev (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:42.563 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-18T14:19:42.831 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-18T14:19:43.141 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-18T14:19:43.494 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-rados (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:43.620 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up librgw2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:43.770 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-rbd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:43.896 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up rbd-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:44.005 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-18T14:19:44.298 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-18T14:19:44.583 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-rgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:44.709 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-cephfs (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:44.835 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-18T14:19:45.797 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:46.333 INFO:teuthology.orchestra.run.smithi046.stdout:Adding group ceph....done 2024-09-18T14:19:46.693 INFO:teuthology.orchestra.run.smithi046.stdout:Adding system user ceph....done 2024-09-18T14:19:46.869 INFO:teuthology.orchestra.run.smithi046.stdout:Setting system user ceph properties....done 2024-09-18T14:19:46.891 INFO:teuthology.orchestra.run.smithi046.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-18T14:19:47.049 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-18T14:19:47.372 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-18T14:19:47.848 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-test (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:47.974 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up radosgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:48.248 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-18T14:19:48.248 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-18T14:19:48.893 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-base (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:49.283 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-18T14:19:49.720 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mds (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:49.968 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-18T14:19:49.968 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-18T14:19:50.429 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:50.688 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-18T14:19:50.688 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-18T14:19:51.189 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-osd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:51.664 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-18T14:19:51.664 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-18T14:19:52.098 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr-k8sevents (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:52.233 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:52.376 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mon (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:52.603 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-18T14:19:52.603 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-18T14:19:53.044 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr-cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:53.220 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:53.346 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr-dashboard (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:53.489 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-volume (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:19:53.615 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-18T14:19:53.750 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-18T14:19:54.035 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-18T14:19:56.929 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-18T14:19:57.535 INFO:teuthology.orchestra.run.smithi046.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-18T14:19:57.541 DEBUG:teuthology.parallel:result is None 2024-09-18T14:19:57.542 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T14:19:57.697 DEBUG:teuthology.orchestra.run.smithi046:> dpkg-query -W -f '${Version}' ceph 2024-09-18T14:19:57.720 INFO:teuthology.orchestra.run.smithi046.stdout:17.2.7-1645-g7061ea6c-1focal 2024-09-18T14:19:57.721 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1645-g7061ea6c-1focal 2024-09-18T14:19:57.721 INFO:teuthology.task.install:The correct ceph version 17.2.7-1645-g7061ea6c-1focal is installed. 2024-09-18T14:19:57.723 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-18T14:19:57.723 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-18T14:19:57.723 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-18T14:19:57.777 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-18T14:19:57.778 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-18T14:19:57.778 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/daemon-helper 2024-09-18T14:19:57.792 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-18T14:19:57.808 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-18T14:19:57.809 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-18T14:19:57.809 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-18T14:19:57.821 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-18T14:19:57.878 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-18T14:19:57.878 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-18T14:19:57.878 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/stdin-killer 2024-09-18T14:19:57.888 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-18T14:19:57.903 INFO:teuthology.run_tasks:Running task workunit... 2024-09-18T14:19:57.920 INFO:tasks.workunit:Pulling workunits from ref 7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T14:19:57.921 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-18T14:19:57.921 INFO:tasks.workunit:timeout=3h 2024-09-18T14:19:57.921 INFO:tasks.workunit:cleanup=True 2024-09-18T14:19:57.921 DEBUG:teuthology.orchestra.run.smithi046:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-18T14:19:57.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T14:19:57.953 INFO:teuthology.orchestra.run.smithi046.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-18T14:19:57.954 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-18T14:19:57.961 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-18T14:19:57.961 DEBUG:teuthology.orchestra.run.smithi046:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-18T14:19:57.966 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T14:19:58.018 INFO:tasks.workunit.client.0.smithi046.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-18T14:20:51.379 INFO:tasks.workunit.client.0.smithi046.stderr:Note: switching to '7061ea6ca50908d104e9f3979dc615e118d9e153'. 2024-09-18T14:20:51.379 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-09-18T14:20:51.379 INFO:tasks.workunit.client.0.smithi046.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-18T14:20:51.380 INFO:tasks.workunit.client.0.smithi046.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-18T14:20:51.380 INFO:tasks.workunit.client.0.smithi046.stderr:state without impacting any branches by switching back to a branch. 2024-09-18T14:20:51.380 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-09-18T14:20:51.380 INFO:tasks.workunit.client.0.smithi046.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-18T14:20:51.380 INFO:tasks.workunit.client.0.smithi046.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-18T14:20:51.380 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-09-18T14:20:51.380 INFO:tasks.workunit.client.0.smithi046.stderr: git switch -c 2024-09-18T14:20:51.380 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-09-18T14:20:51.380 INFO:tasks.workunit.client.0.smithi046.stderr:Or undo this operation with: 2024-09-18T14:20:51.380 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-09-18T14:20:51.380 INFO:tasks.workunit.client.0.smithi046.stderr: git switch - 2024-09-18T14:20:51.380 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-09-18T14:20:51.380 INFO:tasks.workunit.client.0.smithi046.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-18T14:20:51.381 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-09-18T14:20:51.381 INFO:tasks.workunit.client.0.smithi046.stderr:HEAD is now at 7061ea6ca50 Merge branch 'wip-65946-quincy' of https://github.com/amathuria/ceph into wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T14:20:51.384 DEBUG:teuthology.orchestra.run.smithi046:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-18T14:20:51.393 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-09-18T14:20:51.393 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-18T14:20:51.400 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2024-09-18T14:20:51.400 INFO:tasks.workunit:Running workunit scrub/osd-mapper.sh... 2024-09-18T14:20:51.401 DEBUG:teuthology.orchestra.run.smithi046:workunit test scrub/osd-mapper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh 2024-09-18T14:20:51.422 INFO:tasks.workunit.client.0.smithi046.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T14:20:51.438 INFO:tasks.workunit.client.0.smithi046.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T14:20:51.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-18T14:20:51.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-18T14:20:51.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T14:20:51.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T14:20:51.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T14:20:51.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T14:20:51.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T14:20:51.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T14:20:51.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-mapper 2024-09-18T14:20:51.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:11: run: local dir=td/osd-mapper 2024-09-18T14:20:51.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:12: run: shift 2024-09-18T14:20:51.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: export CEPH_MON=127.0.0.1:7144 2024-09-18T14:20:51.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: CEPH_MON=127.0.0.1:7144 2024-09-18T14:20:51.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:15: run: export CEPH_ARGS 2024-09-18T14:20:51.439 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: uuidgen 2024-09-18T14:20:51.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: CEPH_ARGS+='--fsid=ea3ca1de-6959-4a8a-b360-5739e175b9be --auth-supported=none ' 2024-09-18T14:20:51.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:17: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2024-09-18T14:20:51.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:19: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-09-18T14:20:51.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: set 2024-09-18T14:20:51.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T14:20:51.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: local funcs=TEST_truncated_sna_record 2024-09-18T14:20:51.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:21: run: for func in $funcs 2024-09-18T14:20:51.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:22: run: setup td/osd-mapper 2024-09-18T14:20:51.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-mapper 2024-09-18T14:20:51.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-mapper 2024-09-18T14:20:51.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-09-18T14:20:51.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:20:51.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-09-18T14:20:51.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:20:51.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:20:51.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:20:51.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:20:51.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:20:51.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:20:51.450 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:20:51.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:20:51.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:20:51.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:20:51.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:20:51.454 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:20:51.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:20:51.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:20:51.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:20:51.456 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:20:51.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:20:51.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:20:51.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-09-18T14:20:51.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:20:51.462 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:20:51.462 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17204 2024-09-18T14:20:51.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17204 2024-09-18T14:20:51.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:20:51.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:20:51.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-mapper 2024-09-18T14:20:51.465 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:20:51.465 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:20:51.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17204 2024-09-18T14:20:51.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17204 2024-09-18T14:20:51.468 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:20:51.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T14:20:51.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T14:20:51.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:20:51.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-mapper 1' TERM HUP INT 2024-09-18T14:20:51.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:23: run: TEST_truncated_sna_record td/osd-mapper 2024-09-18T14:20:51.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:45: TEST_truncated_sna_record: local dir=td/osd-mapper 2024-09-18T14:20:51.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: cluster_conf=(['osds_num']="3" ['pgs_in_pool']="4" ['pool_name']="test") 2024-09-18T14:20:51.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: local -A cluster_conf 2024-09-18T14:20:51.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:52: TEST_truncated_sna_record: local extr_dbg=1 2024-09-18T14:20:51.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: (( extr_dbg > 1 )) 2024-09-18T14:20:51.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:54: TEST_truncated_sna_record: standard_scrub_cluster td/osd-mapper cluster_conf 2024-09-18T14:20:51.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-mapper 2024-09-18T14:20:51.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-09-18T14:20:51.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-09-18T14:20:51.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=4 2024-09-18T14:20:51.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=test 2024-09-18T14:20:51.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=test 2024-09-18T14:20:51.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local extra_pars= 2024-09-18T14:20:51.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-09-18T14:20:51.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-09-18T14:20:51.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-09-18T14:20:53.532 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 d26d7718-9f0f-4c35-968c-7fcaa7cb790f 2024-09-18T14:20:53.881 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:20:53.981 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:20:53.975+0000 7f7b0d6bf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:20:53.984 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:20:53.979+0000 7f7b0d6bf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:20:53.986 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:20:53.979+0000 7f7b0d6bf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:20:53.987 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:20:53.983+0000 7f7b0d6bf540 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-09-18T14:20:53.987 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:20:53.983+0000 7f7b0d6bf540 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-09-18T14:20:53.987 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:20:53.983+0000 7f7b0d6bf540 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-09-18T14:20:54.032 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:20:54.027+0000 7f7b0d6bf540 -1 bluestore(td/osd-mapper/0) _read_fsid unparsable uuid 2024-09-18T14:20:59.770 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T14:20:59.956 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T14:20:59.981 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:20:59.975+0000 7f67b972f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:20:59.983 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:20:59.979+0000 7f67b972f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:20:59.987 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:20:59.979+0000 7f67b972f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:21:00.188 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:21:00.301 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:21:00.295+0000 7f67b972f540 -1 Falling back to public interface 2024-09-18T14:21:01.396 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:21:02.569 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:21:03.168 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:21:03.163+0000 7f67b972f540 -1 osd.0 0 log_to_monitors true 2024-09-18T14:21:03.875 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:21:05.287 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:21:06.635 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:21:08.030 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:21:09.341 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:21:10.658 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T14:21:12.050 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T14:21:13.356 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T14:21:14.663 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T14:21:16.273 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T14:21:17.760 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T14:21:19.148 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T14:21:20.460 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T14:21:21.872 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T14:21:23.355 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T14:21:24.767 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T14:21:26.191 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T14:21:27.948 INFO:tasks.workunit.client.0.smithi046.stdout:20 2024-09-18T14:21:28.266 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/18241,v1:127.0.0.1:6801/18241] [v2:127.0.0.1:6802/18241,v1:127.0.0.1:6803/18241] exists,up d26d7718-9f0f-4c35-968c-7fcaa7cb790f 2024-09-18T14:21:28.273 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 c12eae97-da58-469e-af2e-a39a17ac70ba 2024-09-18T14:21:28.721 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:21:28.774 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:21:28.767+0000 7fc63e668540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:21:28.803 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:21:28.799+0000 7fc63e668540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:21:28.805 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:21:28.799+0000 7fc63e668540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:21:28.806 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:21:28.799+0000 7fc63e668540 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-09-18T14:21:28.806 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:21:28.799+0000 7fc63e668540 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-09-18T14:21:28.806 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:21:28.799+0000 7fc63e668540 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-09-18T14:21:28.879 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:21:28.875+0000 7fc63e668540 -1 bluestore(td/osd-mapper/1) _read_fsid unparsable uuid 2024-09-18T14:21:34.905 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T14:21:35.290 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T14:21:35.320 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:21:35.316+0000 7f7dac501540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:21:35.323 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:21:35.320+0000 7f7dac501540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:21:35.324 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:21:35.320+0000 7f7dac501540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:21:35.618 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:21:35.885 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:21:35.880+0000 7f7dac501540 -1 Falling back to public interface 2024-09-18T14:21:36.957 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:21:38.268 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:21:38.583 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:21:38.576+0000 7f7dac501540 -1 osd.1 0 log_to_monitors true 2024-09-18T14:21:39.581 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:21:40.941 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:21:42.266 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:21:43.684 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:21:44.994 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:21:46.313 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T14:21:47.645 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T14:21:48.955 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T14:21:50.270 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T14:21:51.712 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T14:21:53.151 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T14:21:54.590 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T14:21:55.947 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T14:21:57.675 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T14:21:59.476 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T14:22:01.162 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T14:22:02.591 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T14:22:02.912 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/20222,v1:127.0.0.1:6811/20222] [v2:127.0.0.1:6812/20222,v1:127.0.0.1:6813/20222] exists,up c12eae97-da58-469e-af2e-a39a17ac70ba 2024-09-18T14:22:02.978 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 651016a2-a525-44f1-87cf-828826a68714 2024-09-18T14:22:03.389 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:22:03.441 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:22:03.437+0000 7fc55d40e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:22:03.447 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:22:03.441+0000 7fc55d40e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:22:03.449 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:22:03.445+0000 7fc55d40e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:22:03.450 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:22:03.445+0000 7fc55d40e540 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-09-18T14:22:03.450 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:22:03.445+0000 7fc55d40e540 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-09-18T14:22:03.450 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:22:03.445+0000 7fc55d40e540 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-09-18T14:22:03.519 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:22:03.517+0000 7fc55d40e540 -1 bluestore(td/osd-mapper/2) _read_fsid unparsable uuid 2024-09-18T14:22:10.007 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T14:22:10.385 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T14:22:10.416 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:22:10.409+0000 7fb110483540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:22:10.419 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:22:10.413+0000 7fb110483540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:22:10.421 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:22:10.417+0000 7fb110483540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:22:10.741 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:22:10.967 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:22:10.961+0000 7fb110483540 -1 Falling back to public interface 2024-09-18T14:22:12.042 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:22:13.350 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:22:13.678 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:22:13.673+0000 7fb110483540 -1 osd.2 0 log_to_monitors true 2024-09-18T14:22:14.660 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:22:16.098 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:22:17.409 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:22:18.720 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:22:20.065 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:22:21.373 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T14:22:22.745 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T14:22:24.057 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T14:22:25.374 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T14:22:26.722 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T14:22:28.120 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T14:22:29.709 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T14:22:31.205 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T14:22:32.882 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T14:22:34.583 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T14:22:35.998 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T14:22:37.633 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T14:22:37.946 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/22027,v1:127.0.0.1:6819/22027] [v2:127.0.0.1:6820/22027,v1:127.0.0.1:6821/22027] exists,up 651016a2-a525-44f1-87cf-828826a68714 2024-09-18T14:22:38.402 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'test' created 2024-09-18T14:22:40.581 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836501 2024-09-18T14:22:42.254 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672971 2024-09-18T14:22:42.593 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509444 2024-09-18T14:22:45.299 INFO:tasks.workunit.client.0.smithi046.stdout:standard_scrub_cluster: dbg: test pool is test 1 2024-09-18T14:22:45.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:55: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2024-09-18T14:22:45.498 INFO:tasks.workunit.client.0.smithi046.stderr:Error ENOENT: error setting 'osd_stats_update_period_not_scrubbing' to '1': (2) No such file or directory 2024-09-18T14:22:45.499 INFO:tasks.workunit.client.0.smithi046.stderr:Error ENOENT: error setting 'osd_stats_update_period_not_scrubbing' to '1': (2) No such file or directory 2024-09-18T14:22:45.499 INFO:tasks.workunit.client.0.smithi046.stderr:Error ENOENT: error setting 'osd_stats_update_period_not_scrubbing' to '1': (2) No such file or directory 2024-09-18T14:22:45.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:56: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_scrubbing 1 2024-09-18T14:22:45.659 INFO:tasks.workunit.client.0.smithi046.stderr:Error ENOENT: error setting 'osd_stats_update_period_scrubbing' to '1': (2) No such file or directory 2024-09-18T14:22:45.659 INFO:tasks.workunit.client.0.smithi046.stderr:Error ENOENT: error setting 'osd_stats_update_period_scrubbing' to '1': (2) No such file or directory 2024-09-18T14:22:45.660 INFO:tasks.workunit.client.0.smithi046.stderr:Error ENOENT: error setting 'osd_stats_update_period_scrubbing' to '1': (2) No such file or directory 2024-09-18T14:22:45.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:58: TEST_truncated_sna_record: local osdn=3 2024-09-18T14:22:45.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:59: TEST_truncated_sna_record: local poolid=1 2024-09-18T14:22:45.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:60: TEST_truncated_sna_record: local poolname=test 2024-09-18T14:22:45.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:61: TEST_truncated_sna_record: local objname=objxxx 2024-09-18T14:22:45.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:64: TEST_truncated_sna_record: make_a_clone test objxxx snap01 snap02 2024-09-18T14:22:45.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-09-18T14:22:45.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-09-18T14:22:45.990 INFO:tasks.workunit.client.0.smithi046.stdout:created pool test snap snap01 2024-09-18T14:22:46.257 INFO:tasks.workunit.client.0.smithi046.stdout:created pool test snap snap02 2024-09-18T14:22:46.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:65: TEST_truncated_sna_record: make_a_clone test objxxx snap13 2024-09-18T14:22:46.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-09-18T14:22:46.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-09-18T14:22:46.761 INFO:tasks.workunit.client.0.smithi046.stdout:created pool test snap snap13 2024-09-18T14:22:46.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:66: TEST_truncated_sna_record: make_a_clone test objxxx snap24 snap25 2024-09-18T14:22:46.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-09-18T14:22:46.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-09-18T14:22:47.221 INFO:tasks.workunit.client.0.smithi046.stdout:created pool test snap snap24 2024-09-18T14:22:47.426 INFO:tasks.workunit.client.0.smithi046.stdout:created pool test snap snap25 2024-09-18T14:22:47.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: echo 4384 2024-09-18T14:22:47.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: rados -p test put objxxx - 2024-09-18T14:22:47.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-09-18T14:22:47.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: jq -r .pgid 2024-09-18T14:22:48.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: local pgid=1.3 2024-09-18T14:22:48.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-09-18T14:22:48.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: jq -r '.up[0]' 2024-09-18T14:22:48.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: local osd=1 2024-09-18T14:22:48.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:72: TEST_truncated_sna_record: echo 'pgid is 1.3 (primary: osd.1)' 2024-09-18T14:22:48.356 INFO:tasks.workunit.client.0.smithi046.stdout:pgid is 1.3 (primary: osd.1) 2024-09-18T14:22:48.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:74: TEST_truncated_sna_record: set_query_debug 1.3 2024-09-18T14:22:48.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:293: set_query_debug: local pgid=1.3 2024-09-18T14:22:48.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:295: set_query_debug: ceph pg dump pgs_brief 2024-09-18T14:22:48.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:295: set_query_debug: awk -v 'pg=^1.3' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2024-09-18T14:22:48.688 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs_brief 2024-09-18T14:22:48.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:295: set_query_debug: local prim_osd=1 2024-09-18T14:22:48.704 INFO:tasks.workunit.client.0.smithi046.stdout:Setting scrub debug data. Primary for 1.3 is 1 2024-09-18T14:22:48.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: echo 'Setting scrub debug data. Primary for 1.3 is 1' 2024-09-18T14:22:48.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:298: set_query_debug: get_asok_path osd.1 2024-09-18T14:22:48.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:22:48.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:22:48.705 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:22:48.705 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:22:48.705 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17204 2024-09-18T14:22:48.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17204/ceph-osd.1.asok 2024-09-18T14:22:48.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:298: set_query_debug: CEPH_ARGS= 2024-09-18T14:22:48.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:298: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.17204/ceph-osd.1.asok scrubdebug 1.3 set sessions 2024-09-18T14:22:48.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-18T14:22:48.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-09-18T14:22:48.842 INFO:tasks.workunit.client.0.smithi046.stdout:{"success":true}{ 2024-09-18T14:22:48.842 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "objxxx", 2024-09-18T14:22:48.842 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T14:22:48.842 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T14:22:48.842 INFO:tasks.workunit.client.0.smithi046.stdout: "id": 2, 2024-09-18T14:22:48.843 INFO:tasks.workunit.client.0.smithi046.stdout: "snapshots": [ 2024-09-18T14:22:48.843 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T14:22:48.843 INFO:tasks.workunit.client.0.smithi046.stdout: "id": 1, 2024-09-18T14:22:48.843 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "snap01" 2024-09-18T14:22:48.843 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T14:22:48.843 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T14:22:48.843 INFO:tasks.workunit.client.0.smithi046.stdout: "id": 2, 2024-09-18T14:22:48.843 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "snap02" 2024-09-18T14:22:48.843 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T14:22:48.843 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T14:22:48.843 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 6, 2024-09-18T14:22:48.843 INFO:tasks.workunit.client.0.smithi046.stdout: "overlaps": [] 2024-09-18T14:22:48.843 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T14:22:48.843 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T14:22:48.844 INFO:tasks.workunit.client.0.smithi046.stdout: "id": 3, 2024-09-18T14:22:48.844 INFO:tasks.workunit.client.0.smithi046.stdout: "snapshots": [ 2024-09-18T14:22:48.844 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T14:22:48.844 INFO:tasks.workunit.client.0.smithi046.stdout: "id": 3, 2024-09-18T14:22:48.844 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "snap13" 2024-09-18T14:22:48.844 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T14:22:48.844 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T14:22:48.844 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 6, 2024-09-18T14:22:48.844 INFO:tasks.workunit.client.0.smithi046.stdout: "overlaps": [] 2024-09-18T14:22:48.844 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T14:22:48.844 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T14:22:48.844 INFO:tasks.workunit.client.0.smithi046.stdout: "id": 5, 2024-09-18T14:22:48.844 INFO:tasks.workunit.client.0.smithi046.stdout: "snapshots": [ 2024-09-18T14:22:48.844 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T14:22:48.845 INFO:tasks.workunit.client.0.smithi046.stdout: "id": 4, 2024-09-18T14:22:48.845 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "snap24" 2024-09-18T14:22:48.845 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T14:22:48.845 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T14:22:48.845 INFO:tasks.workunit.client.0.smithi046.stdout: "id": 5, 2024-09-18T14:22:48.845 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "snap25" 2024-09-18T14:22:48.845 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T14:22:48.845 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T14:22:48.845 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 5, 2024-09-18T14:22:48.845 INFO:tasks.workunit.client.0.smithi046.stdout: "overlaps": [] 2024-09-18T14:22:48.845 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T14:22:48.845 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T14:22:48.845 INFO:tasks.workunit.client.0.smithi046.stdout: "id": "head", 2024-09-18T14:22:48.845 INFO:tasks.workunit.client.0.smithi046.stdout: "snapshots": [], 2024-09-18T14:22:48.846 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 5 2024-09-18T14:22:48.846 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T14:22:48.846 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T14:22:48.846 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:22:48.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:80: TEST_truncated_sna_record: ceph pg 1.3 deep_scrub 2024-09-18T14:22:48.964 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:22:48.964 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": true, 2024-09-18T14:22:48.964 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:21:08.963294+0000" 2024-09-18T14:22:48.964 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:22:48.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:84: TEST_truncated_sna_record: sleep 3 2024-09-18T14:22:51.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:85: TEST_truncated_sna_record: ceph pg dump pgs 2024-09-18T14:22:52.274 INFO:tasks.workunit.client.0.smithi046.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:22:52.274 INFO:tasks.workunit.client.0.smithi046.stdout:1.3 4 0 0 0 0 22 0 0 7 0 7 active+clean 2024-09-18T14:22:39.581077+0000 25'7 25:19 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:22:38.295459+0000 0'0 2024-09-18T14:22:38.295459+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:22:38.295459+0000 0 0 2024-09-18T14:22:52.275 INFO:tasks.workunit.client.0.smithi046.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:22:39.581258+0000 0'0 23:14 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-18T14:22:38.295459+0000 0'0 2024-09-18T14:22:38.295459+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:22:38.295459+0000 0 0 2024-09-18T14:22:52.275 INFO:tasks.workunit.client.0.smithi046.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:22:39.581484+0000 0'0 22:13 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T14:22:38.295459+0000 0'0 2024-09-18T14:22:38.295459+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:22:38.295459+0000 0 0 2024-09-18T14:22:52.275 INFO:tasks.workunit.client.0.smithi046.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:22:39.581109+0000 0'0 24:15 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:22:38.295459+0000 0'0 2024-09-18T14:22:38.295459+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:22:38.295459+0000 0 0 2024-09-18T14:22:52.275 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-09-18T14:22:52.275 INFO:tasks.workunit.client.0.smithi046.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:22:52.278 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:22:52.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:86: TEST_truncated_sna_record: grep -a -q -- 'event: --^^^^---- ScrubFinished' td/osd-mapper/osd.1.log 2024-09-18T14:22:52.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:90: TEST_truncated_sna_record: ceph pg dump pgs 2024-09-18T14:22:52.578 INFO:tasks.workunit.client.0.smithi046.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:22:52.578 INFO:tasks.workunit.client.0.smithi046.stdout:1.3 4 0 0 0 0 22 0 0 7 0 7 active+clean 2024-09-18T14:22:39.581077+0000 25'7 25:19 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:22:38.295459+0000 0'0 2024-09-18T14:22:38.295459+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:22:38.295459+0000 0 0 2024-09-18T14:22:52.578 INFO:tasks.workunit.client.0.smithi046.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:22:39.581258+0000 0'0 23:14 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-18T14:22:38.295459+0000 0'0 2024-09-18T14:22:38.295459+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:22:38.295459+0000 0 0 2024-09-18T14:22:52.578 INFO:tasks.workunit.client.0.smithi046.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:22:39.581484+0000 0'0 22:13 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T14:22:38.295459+0000 0'0 2024-09-18T14:22:38.295459+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:22:38.295459+0000 0 0 2024-09-18T14:22:52.578 INFO:tasks.workunit.client.0.smithi046.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:22:39.581109+0000 0'0 24:15 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:22:38.295459+0000 0'0 2024-09-18T14:22:38.295459+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:22:38.295459+0000 0 0 2024-09-18T14:22:52.579 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-09-18T14:22:52.579 INFO:tasks.workunit.client.0.smithi046.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:22:52.581 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:22:52.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:91: TEST_truncated_sna_record: ceph osd set noscrub 2024-09-18T14:22:53.033 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T14:22:53.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:92: TEST_truncated_sna_record: ceph osd set nodeep-scrub 2024-09-18T14:22:53.510 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T14:22:53.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:93: TEST_truncated_sna_record: sleep 5 2024-09-18T14:22:58.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:94: TEST_truncated_sna_record: grep -a -q -v ERR td/osd-mapper/osd.1.log 2024-09-18T14:22:58.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:97: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-09-18T14:22:58.528 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:22:58.528 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:22:58.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:22:58.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:22:58.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:22:58.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:22:58.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-18T14:22:58.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-18T14:22:58.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-18T14:22:58.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-18T14:22:58.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-09-18T14:22:58.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-09-18T14:22:58.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-09-18T14:22:58.847 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: wc -l 2024-09-18T14:23:00.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: local num_sna_b4=5 2024-09-18T14:23:00.661 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: expr 3 - 1 2024-09-18T14:23:00.662 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: seq 0 2 2024-09-18T14:23:00.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-18T14:23:00.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/0 2024-09-18T14:23:00.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0)' 2024-09-18T14:23:00.664 INFO:tasks.workunit.client.0.smithi046.stdout:corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0) 2024-09-18T14:23:00.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-09-18T14:23:00.665 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-09-18T14:23:00.665 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-09-18T14:23:00.665 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-09-18T14:23:02.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-18T14:23:02.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-18T14:23:02.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-09-18T14:23:02.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-09-18T14:23:02.490 INFO:tasks.workunit.client.0.smithi046.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-18T14:23:02.491 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-09-18T14:23:02.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.I2AiJYto6H_the_val 2024-09-18T14:23:02.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-18T14:23:02.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.I2AiJYto6H_the_val' 2024-09-18T14:23:02.493 INFO:tasks.workunit.client.0.smithi046.stdout:Value dumped in: /tmp/tmp.I2AiJYto6H_the_val 2024-09-18T14:23:02.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.I2AiJYto6H_the_val 2024-09-18T14:23:03.279 INFO:tasks.workunit.client.0.smithi046.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-09-18T14:23:03.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-18T14:23:03.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-09-18T14:23:03.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-18T14:23:05.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.I2AiJYto6H_the_val 2024-09-18T14:23:06.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-18T14:23:06.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-18T14:23:06.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-09-18T14:23:06.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1)' 2024-09-18T14:23:06.454 INFO:tasks.workunit.client.0.smithi046.stdout:corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1) 2024-09-18T14:23:06.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-09-18T14:23:06.455 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-09-18T14:23:06.455 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-09-18T14:23:06.455 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-09-18T14:23:07.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-18T14:23:07.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-18T14:23:07.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-09-18T14:23:07.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-09-18T14:23:07.519 INFO:tasks.workunit.client.0.smithi046.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-18T14:23:07.520 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-09-18T14:23:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.ebQbm8aw6S_the_val 2024-09-18T14:23:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-18T14:23:07.523 INFO:tasks.workunit.client.0.smithi046.stdout:Value dumped in: /tmp/tmp.ebQbm8aw6S_the_val 2024-09-18T14:23:07.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.ebQbm8aw6S_the_val' 2024-09-18T14:23:07.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.ebQbm8aw6S_the_val 2024-09-18T14:23:08.304 INFO:tasks.workunit.client.0.smithi046.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-09-18T14:23:08.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-18T14:23:08.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-09-18T14:23:08.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-18T14:23:10.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.ebQbm8aw6S_the_val 2024-09-18T14:23:11.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-18T14:23:11.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-18T14:23:11.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/2 2024-09-18T14:23:11.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2)' 2024-09-18T14:23:11.477 INFO:tasks.workunit.client.0.smithi046.stdout:corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2) 2024-09-18T14:23:11.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-09-18T14:23:11.478 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-09-18T14:23:11.478 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-09-18T14:23:11.478 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-09-18T14:23:13.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-18T14:23:13.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-18T14:23:13.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-09-18T14:23:13.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-09-18T14:23:13.297 INFO:tasks.workunit.client.0.smithi046.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-18T14:23:13.299 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-09-18T14:23:13.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.DadM0PQAtZ_the_val 2024-09-18T14:23:13.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-18T14:23:13.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.DadM0PQAtZ_the_val' 2024-09-18T14:23:13.301 INFO:tasks.workunit.client.0.smithi046.stdout:Value dumped in: /tmp/tmp.DadM0PQAtZ_the_val 2024-09-18T14:23:13.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.DadM0PQAtZ_the_val 2024-09-18T14:23:14.106 INFO:tasks.workunit.client.0.smithi046.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-09-18T14:23:14.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-18T14:23:14.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-09-18T14:23:14.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-18T14:23:15.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.DadM0PQAtZ_the_val 2024-09-18T14:23:17.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-18T14:23:17.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:129: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off ' 2024-09-18T14:23:17.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: echo --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off 2024-09-18T14:23:17.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off' 2024-09-18T14:23:17.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-18T14:23:17.251 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: expr 3 - 1 2024-09-18T14:23:17.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: seq 0 2 2024-09-18T14:23:17.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-18T14:23:17.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=ea3ca1de-6959-4a8a-b360-5739e175b9be --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off' 2024-09-18T14:23:17.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 0 2024-09-18T14:23:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-09-18T14:23:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:23:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T14:23:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:23:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/0 2024-09-18T14:23:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=ea3ca1de-6959-4a8a-b360-5739e175b9be --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off' 2024-09-18T14:23:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:23:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:23:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:23:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/0' 2024-09-18T14:23:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/0/journal' 2024-09-18T14:23:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:23:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:23:17.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-09-18T14:23:17.257 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:23:17.258 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:23:17.258 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:23:17.258 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:23:17.258 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:23:17.258 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17204 2024-09-18T14:23:17.258 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17204/$cluster-$name.asok' 2024-09-18T14:23:17.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17204/$cluster-$name.asok' 2024-09-18T14:23:17.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:23:17.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-09-18T14:23:17.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-09-18T14:23:17.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:23:17.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:23:17.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:23:17.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:23:17.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:23:17.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:23:17.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/0 2024-09-18T14:23:17.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T14:23:17.261 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T14:23:17.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=ea3ca1de-6959-4a8a-b360-5739e175b9be --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/0 --osd-journal=td/osd-mapper/0/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.17204/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:23:17.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/0/whoami 2024-09-18T14:23:17.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T14:23:17.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:23:17.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:23:17.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:23:17.299 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:23:17.292+0000 7f621ab7c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:23:17.303 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:23:17.296+0000 7f621ab7c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:23:17.305 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:23:17.300+0000 7f621ab7c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:23:17.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T14:23:17.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:23:17.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:23:17.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:23:17.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:23:17.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:17.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:23:17.578 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:23:17.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:17.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:23:17.839 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:23:17.832+0000 7f621ab7c540 -1 Falling back to public interface 2024-09-18T14:23:17.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:23:18.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:23:18.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:18.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:23:18.931 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:23:18.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:18.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:23:19.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:23:20.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:23:20.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:20.249 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:23:20.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:23:20.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:20.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:23:20.545 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:23:20.540+0000 7f621ab7c540 -1 osd.0 27 log_to_monitors true 2024-09-18T14:23:20.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:23:21.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:23:21.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:21.585 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:23:21.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:23:21.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:21.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:23:21.920 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 30 up_thru 30 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6800/28793,v1:127.0.0.1:6801/28793] [v2:127.0.0.1:6802/28793,v1:127.0.0.1:6803/28793] exists,up d26d7718-9f0f-4c35-968c-7fcaa7cb790f 2024-09-18T14:23:21.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:23:21.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:23:21.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:23:21.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-18T14:23:21.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=ea3ca1de-6959-4a8a-b360-5739e175b9be --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off' 2024-09-18T14:23:21.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 1 2024-09-18T14:23:21.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-09-18T14:23:21.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:23:21.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T14:23:21.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:23:21.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/1 2024-09-18T14:23:21.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=ea3ca1de-6959-4a8a-b360-5739e175b9be --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off' 2024-09-18T14:23:21.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:23:21.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:23:21.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:23:21.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/1' 2024-09-18T14:23:21.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/1/journal' 2024-09-18T14:23:21.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:23:21.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:23:21.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-09-18T14:23:21.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:23:21.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:23:21.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:23:21.929 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:23:21.929 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:23:21.929 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17204 2024-09-18T14:23:21.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17204/$cluster-$name.asok' 2024-09-18T14:23:21.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17204/$cluster-$name.asok' 2024-09-18T14:23:21.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:23:21.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-09-18T14:23:21.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-09-18T14:23:21.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:23:21.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:23:21.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:23:21.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:23:21.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:23:21.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:23:21.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/1 2024-09-18T14:23:21.931 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T14:23:21.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T14:23:21.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=ea3ca1de-6959-4a8a-b360-5739e175b9be --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/1 --osd-journal=td/osd-mapper/1/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.17204/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:23:21.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/1/whoami 2024-09-18T14:23:21.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T14:23:21.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:23:21.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:23:21.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:23:21.964 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:23:21.960+0000 7f8d39871540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:23:21.973 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:23:21.968+0000 7f8d39871540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:23:21.974 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:23:21.968+0000 7f8d39871540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:23:22.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T14:23:22.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:23:22.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:23:22.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:23:22.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:23:22.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:22.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:23:22.282 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:23:22.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:22.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:23:22.507 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:23:22.500+0000 7f8d39871540 -1 Falling back to public interface 2024-09-18T14:23:22.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:23:23.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:23:23.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:23.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:23:23.615 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:23:23.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:23.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:23:23.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:23:24.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:23:24.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:24.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:23:24.942 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:23:24.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:24.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:23:25.196 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:23:25.188+0000 7f8d39871540 -1 osd.1 27 log_to_monitors true 2024-09-18T14:23:25.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:23:26.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:23:26.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:26.296 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:23:26.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:23:26.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:26.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:23:26.610 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 33 up_thru 33 down_at 28 last_clean_interval [10,27) [v2:127.0.0.1:6810/29371,v1:127.0.0.1:6811/29371] [v2:127.0.0.1:6812/29371,v1:127.0.0.1:6813/29371] exists,up c12eae97-da58-469e-af2e-a39a17ac70ba 2024-09-18T14:23:26.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:23:26.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:23:26.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:23:26.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-18T14:23:26.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=ea3ca1de-6959-4a8a-b360-5739e175b9be --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off' 2024-09-18T14:23:26.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 2 2024-09-18T14:23:26.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-09-18T14:23:26.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:23:26.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T14:23:26.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:23:26.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/2 2024-09-18T14:23:26.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=ea3ca1de-6959-4a8a-b360-5739e175b9be --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off' 2024-09-18T14:23:26.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:23:26.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:23:26.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:23:26.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/2' 2024-09-18T14:23:26.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/2/journal' 2024-09-18T14:23:26.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:23:26.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:23:26.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-09-18T14:23:26.614 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:23:26.614 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:23:26.614 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:23:26.614 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:23:26.614 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:23:26.614 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17204 2024-09-18T14:23:26.616 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17204/$cluster-$name.asok' 2024-09-18T14:23:26.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17204/$cluster-$name.asok' 2024-09-18T14:23:26.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:23:26.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-09-18T14:23:26.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-09-18T14:23:26.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:23:26.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:23:26.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:23:26.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:23:26.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:23:26.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:23:26.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/2 2024-09-18T14:23:26.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T14:23:26.617 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T14:23:26.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=ea3ca1de-6959-4a8a-b360-5739e175b9be --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/2 --osd-journal=td/osd-mapper/2/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.17204/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:23:26.618 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/2/whoami 2024-09-18T14:23:26.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T14:23:26.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:23:26.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:23:26.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:23:26.653 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:23:26.648+0000 7f6a1ccc4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:23:26.656 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:23:26.648+0000 7f6a1ccc4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:23:26.659 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:23:26.652+0000 7f6a1ccc4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:23:26.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T14:23:26.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:23:26.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:23:26.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:23:26.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:23:26.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:26.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:23:26.924 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:23:26.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:26.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:23:27.195 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:23:27.188+0000 7f6a1ccc4540 -1 Falling back to public interface 2024-09-18T14:23:27.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:23:28.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:23:28.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:28.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:23:28.265 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:23:28.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:28.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:23:28.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:23:29.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:23:29.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:29.592 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:23:29.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:23:29.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:29.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:23:29.800 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:23:29.792+0000 7f6a1ccc4540 -1 osd.2 27 log_to_monitors true 2024-09-18T14:23:29.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:23:30.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:23:30.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:30.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:23:30.910 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:23:30.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:30.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:23:31.393 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 36 up_thru 36 down_at 28 last_clean_interval [15,27) [v2:127.0.0.1:6818/29949,v1:127.0.0.1:6819/29949] [v2:127.0.0.1:6820/29949,v1:127.0.0.1:6821/29949] exists,up 651016a2-a525-44f1-87cf-828826a68714 2024-09-18T14:23:31.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:23:31.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:23:31.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:23:31.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:136: TEST_truncated_sna_record: sleep 1 2024-09-18T14:23:32.397 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: expr 3 - 1 2024-09-18T14:23:32.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: seq 0 2 2024-09-18T14:23:32.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-18T14:23:32.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.0 version 2024-09-18T14:23:32.532 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:23:32.533 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "17.2.7-1645-g7061ea6c", 2024-09-18T14:23:32.533 INFO:tasks.workunit.client.0.smithi046.stdout: "release": "quincy", 2024-09-18T14:23:32.533 INFO:tasks.workunit.client.0.smithi046.stdout: "release_type": "stable" 2024-09-18T14:23:32.533 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:23:32.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-18T14:23:32.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.1 version 2024-09-18T14:23:32.722 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:23:32.722 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "17.2.7-1645-g7061ea6c", 2024-09-18T14:23:32.722 INFO:tasks.workunit.client.0.smithi046.stdout: "release": "quincy", 2024-09-18T14:23:32.722 INFO:tasks.workunit.client.0.smithi046.stdout: "release_type": "stable" 2024-09-18T14:23:32.722 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:23:32.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-18T14:23:32.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.2 version 2024-09-18T14:23:32.864 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:23:32.865 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "17.2.7-1645-g7061ea6c", 2024-09-18T14:23:32.865 INFO:tasks.workunit.client.0.smithi046.stdout: "release": "quincy", 2024-09-18T14:23:32.865 INFO:tasks.workunit.client.0.smithi046.stdout: "release_type": "stable" 2024-09-18T14:23:32.865 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:23:32.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:142: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-09-18T14:23:32.915 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:23:32.915 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "objxxx", 2024-09-18T14:23:32.916 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T14:23:32.916 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T14:23:32.916 INFO:tasks.workunit.client.0.smithi046.stdout: "id": 2, 2024-09-18T14:23:32.916 INFO:tasks.workunit.client.0.smithi046.stdout: "snapshots": [ 2024-09-18T14:23:32.916 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T14:23:32.916 INFO:tasks.workunit.client.0.smithi046.stdout: "id": 1, 2024-09-18T14:23:32.916 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "snap01" 2024-09-18T14:23:32.916 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T14:23:32.916 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T14:23:32.916 INFO:tasks.workunit.client.0.smithi046.stdout: "id": 2, 2024-09-18T14:23:32.916 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "snap02" 2024-09-18T14:23:32.916 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T14:23:32.916 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T14:23:32.917 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 6, 2024-09-18T14:23:32.917 INFO:tasks.workunit.client.0.smithi046.stdout: "overlaps": [] 2024-09-18T14:23:32.917 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T14:23:32.917 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T14:23:32.917 INFO:tasks.workunit.client.0.smithi046.stdout: "id": 3, 2024-09-18T14:23:32.917 INFO:tasks.workunit.client.0.smithi046.stdout: "snapshots": [ 2024-09-18T14:23:32.917 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T14:23:32.917 INFO:tasks.workunit.client.0.smithi046.stdout: "id": 3, 2024-09-18T14:23:32.917 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "snap13" 2024-09-18T14:23:32.917 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T14:23:32.917 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T14:23:32.917 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 6, 2024-09-18T14:23:32.917 INFO:tasks.workunit.client.0.smithi046.stdout: "overlaps": [] 2024-09-18T14:23:32.918 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T14:23:32.918 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T14:23:32.918 INFO:tasks.workunit.client.0.smithi046.stdout: "id": 5, 2024-09-18T14:23:32.918 INFO:tasks.workunit.client.0.smithi046.stdout: "snapshots": [ 2024-09-18T14:23:32.918 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T14:23:32.918 INFO:tasks.workunit.client.0.smithi046.stdout: "id": 4, 2024-09-18T14:23:32.918 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "snap24" 2024-09-18T14:23:32.918 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T14:23:32.918 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T14:23:32.918 INFO:tasks.workunit.client.0.smithi046.stdout: "id": 5, 2024-09-18T14:23:32.918 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "snap25" 2024-09-18T14:23:32.918 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T14:23:32.918 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T14:23:32.919 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 5, 2024-09-18T14:23:32.919 INFO:tasks.workunit.client.0.smithi046.stdout: "overlaps": [] 2024-09-18T14:23:32.919 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T14:23:32.919 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T14:23:32.919 INFO:tasks.workunit.client.0.smithi046.stdout: "id": "head", 2024-09-18T14:23:32.919 INFO:tasks.workunit.client.0.smithi046.stdout: "snapshots": [], 2024-09-18T14:23:32.919 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 5 2024-09-18T14:23:32.919 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T14:23:32.919 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T14:23:32.919 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:23:32.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:145: TEST_truncated_sna_record: ceph osd unset nodeep-scrub 2024-09-18T14:23:33.350 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is unset 2024-09-18T14:23:33.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:146: TEST_truncated_sna_record: ceph osd unset noscrub 2024-09-18T14:23:33.870 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is unset 2024-09-18T14:23:33.886 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-09-18T14:23:33.886 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: jq -r '.up[0]' 2024-09-18T14:23:34.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: local cur_prim=1 2024-09-18T14:23:34.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:150: TEST_truncated_sna_record: ceph pg dump pgs 2024-09-18T14:23:34.503 INFO:tasks.workunit.client.0.smithi046.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:23:34.503 INFO:tasks.workunit.client.0.smithi046.stdout:1.3 4 0 0 0 0 22 0 0 7 0 7 peering 2024-09-18T14:23:30.308479+0000 25'7 36:43 [1,2,0] 1 [1,2,0] 1 25'7 2024-09-18T14:22:49.707507+0000 25'7 2024-09-18T14:22:49.707507+0000 0 1 periodic scrub scheduled @ 2024-09-19T14:22:49.707507+0000 4 0 2024-09-18T14:23:34.503 INFO:tasks.workunit.client.0.smithi046.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 peering 2024-09-18T14:23:30.400842+0000 0'0 36:31 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-18T14:22:38.295459+0000 0'0 2024-09-18T14:22:38.295459+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:22:38.295459+0000 0 0 2024-09-18T14:23:34.503 INFO:tasks.workunit.client.0.smithi046.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 peering 2024-09-18T14:23:25.714696+0000 0'0 33:13 [0,1] 0 [0,1] 0 0'0 2024-09-18T14:22:38.295459+0000 0'0 2024-09-18T14:22:38.295459+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:22:38.295459+0000 0 0 2024-09-18T14:23:34.503 INFO:tasks.workunit.client.0.smithi046.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 peering 2024-09-18T14:23:30.308338+0000 0'0 36:23 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:22:38.295459+0000 0'0 2024-09-18T14:22:38.295459+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:22:38.295459+0000 0 0 2024-09-18T14:23:34.503 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-09-18T14:23:34.504 INFO:tasks.workunit.client.0.smithi046.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:23:34.506 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:23:34.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:151: TEST_truncated_sna_record: sleep 2 2024-09-18T14:23:36.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:152: TEST_truncated_sna_record: ceph pg 1.3 deep_scrub 2024-09-18T14:23:36.659 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:23:36.659 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": true, 2024-09-18T14:23:36.659 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:21:56.658068+0000" 2024-09-18T14:23:36.659 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:23:36.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:153: TEST_truncated_sna_record: sleep 5 2024-09-18T14:23:41.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:154: TEST_truncated_sna_record: ceph pg dump pgs 2024-09-18T14:23:41.961 INFO:tasks.workunit.client.0.smithi046.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:23:41.961 INFO:tasks.workunit.client.0.smithi046.stdout:1.3 4 0 0 0 0 22 0 0 7 0 7 active+clean 2024-09-18T14:23:37.221617+0000 25'7 40:67 [1,2,0] 1 [1,2,0] 1 25'7 2024-09-18T14:23:37.221438+0000 25'7 2024-09-18T14:23:37.221438+0000 0 1 periodic scrub scheduled @ 2024-09-19T14:23:37.221438+0000 4 0 2024-09-18T14:23:41.961 INFO:tasks.workunit.client.0.smithi046.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:23:31.615110+0000 0'0 39:42 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-18T14:22:38.295459+0000 0'0 2024-09-18T14:22:38.295459+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:22:38.295459+0000 0 0 2024-09-18T14:23:41.962 INFO:tasks.workunit.client.0.smithi046.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:23:31.615081+0000 0'0 38:26 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T14:22:38.295459+0000 0'0 2024-09-18T14:22:38.295459+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:22:38.295459+0000 0 0 2024-09-18T14:23:41.962 INFO:tasks.workunit.client.0.smithi046.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:23:30.903061+0000 0'0 38:34 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:22:38.295459+0000 0'0 2024-09-18T14:22:38.295459+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:22:38.295459+0000 0 0 2024-09-18T14:23:41.962 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-09-18T14:23:41.962 INFO:tasks.workunit.client.0.smithi046.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:23:41.965 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:23:41.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-18T14:23:41.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-09-18T14:23:41.977 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:23:37.216+0000 7f8d1bd81700 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-09-18T14:23:41.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:156: TEST_truncated_sna_record: grep -a -q ERR td/osd-mapper/osd.1.log 2024-09-18T14:23:41.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-09-18T14:23:41.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: wc -l 2024-09-18T14:23:41.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: local prev_err_cnt=1 2024-09-18T14:23:41.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:161: TEST_truncated_sna_record: echo 'prev count: 1' 2024-09-18T14:23:41.986 INFO:tasks.workunit.client.0.smithi046.stdout:prev count: 1 2024-09-18T14:23:41.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:164: TEST_truncated_sna_record: ceph pg 1.3 deep_scrub 2024-09-18T14:23:42.105 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:23:42.105 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": true, 2024-09-18T14:23:42.105 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:22:02.104536+0000" 2024-09-18T14:23:42.105 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:23:42.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:165: TEST_truncated_sna_record: sleep 5 2024-09-18T14:23:47.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:166: TEST_truncated_sna_record: ceph pg dump pgs 2024-09-18T14:23:47.405 INFO:tasks.workunit.client.0.smithi046.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:23:47.406 INFO:tasks.workunit.client.0.smithi046.stdout:1.3 4 0 0 0 0 22 0 0 7 0 7 active+clean 2024-09-18T14:23:42.162259+0000 25'7 40:79 [1,2,0] 1 [1,2,0] 1 25'7 2024-09-18T14:23:42.162090+0000 25'7 2024-09-18T14:23:42.162090+0000 0 1 periodic scrub scheduled @ 2024-09-19T14:23:42.162090+0000 4 0 2024-09-18T14:23:47.406 INFO:tasks.workunit.client.0.smithi046.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:23:31.615110+0000 0'0 39:42 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-18T14:22:38.295459+0000 0'0 2024-09-18T14:22:38.295459+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:22:38.295459+0000 0 0 2024-09-18T14:23:47.406 INFO:tasks.workunit.client.0.smithi046.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:23:31.615081+0000 0'0 38:26 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T14:22:38.295459+0000 0'0 2024-09-18T14:22:38.295459+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:22:38.295459+0000 0 0 2024-09-18T14:23:47.406 INFO:tasks.workunit.client.0.smithi046.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:23:30.903061+0000 0'0 38:34 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:22:38.295459+0000 0'0 2024-09-18T14:22:38.295459+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:22:38.295459+0000 0 0 2024-09-18T14:23:47.406 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-09-18T14:23:47.406 INFO:tasks.workunit.client.0.smithi046.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:23:47.409 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:23:47.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-18T14:23:47.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-09-18T14:23:47.421 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:23:37.216+0000 7f8d1bd81700 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-09-18T14:23:47.423 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-09-18T14:23:47.423 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: wc -l 2024-09-18T14:23:47.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: local current_err_cnt=1 2024-09-18T14:23:47.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-18T14:23:47.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: echo 'current count: 1' 2024-09-18T14:23:47.427 INFO:tasks.workunit.client.0.smithi046.stdout:current count: 1 2024-09-18T14:23:47.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:170: TEST_truncated_sna_record: (( current_err_cnt == prev_err_cnt )) 2024-09-18T14:23:47.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:171: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-09-18T14:23:47.428 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:23:47.428 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:23:47.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:23:47.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:23:47.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:23:47.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:23:47.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:172: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-09-18T14:23:47.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-18T14:23:47.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-09-18T14:23:47.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-09-18T14:23:47.744 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-09-18T14:23:47.744 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: wc -l 2024-09-18T14:23:49.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: local num_sna_full=5 2024-09-18T14:23:49.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:177: TEST_truncated_sna_record: (( num_sna_full == num_sna_b4 )) 2024-09-18T14:23:49.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:178: TEST_truncated_sna_record: return 0 2024-09-18T14:23:49.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:24: run: teardown td/osd-mapper 2024-09-18T14:23:49.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-09-18T14:23:49.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:23:49.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-09-18T14:23:49.808 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:23:49.808 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:23:49.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:23:49.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:23:49.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:23:49.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:23:49.925 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:23:49.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:23:49.928 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:23:49.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:23:49.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:23:49.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:23:49.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:23:49.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:23:49.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:23:49.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:23:49.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:23:49.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:23:49.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-09-18T14:23:49.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:23:49.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:23:49.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17204 2024-09-18T14:23:49.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17204 2024-09-18T14:23:49.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:23:49.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:23:49.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T14:23:49.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-mapper 0 2024-09-18T14:23:49.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-09-18T14:23:49.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T14:23:49.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-09-18T14:23:49.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:23:49.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:23:49.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:23:49.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:23:49.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:23:49.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:23:49.969 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:23:49.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:23:49.971 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:23:49.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:23:49.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:23:49.973 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:23:49.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:23:49.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:23:49.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:23:49.976 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:23:49.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:23:49.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T14:23:49.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-09-18T14:23:49.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:23:49.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:23:49.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17204 2024-09-18T14:23:49.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17204 2024-09-18T14:23:49.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:23:49.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:23:49.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T14:23:49.985 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T14:23:49.985 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T14:23:50.042 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2024-09-18T14:23:50.043 DEBUG:teuthology.orchestra.run.smithi046:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2024-09-18T14:23:50.060 INFO:tasks.workunit.client.0.smithi046.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T14:23:50.075 INFO:tasks.workunit.client.0.smithi046.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T14:23:50.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-18T14:23:50.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-18T14:23:50.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T14:23:50.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T14:23:50.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T14:23:50.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T14:23:50.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T14:23:50.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T14:23:50.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-recovery-scrub 2024-09-18T14:23:50.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2024-09-18T14:23:50.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2024-09-18T14:23:50.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2024-09-18T14:23:50.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2024-09-18T14:23:50.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2024-09-18T14:23:50.076 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2024-09-18T14:23:50.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none ' 2024-09-18T14:23:50.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-09-18T14:23:50.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:28: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-09-18T14:23:50.077 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: set 2024-09-18T14:23:50.077 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T14:23:50.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: local 'funcs=TEST_recovery_scrub_1 2024-09-18T14:23:50.081 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_recovery_scrub_2' 2024-09-18T14:23:50.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: for func in $funcs 2024-09-18T14:23:50.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: TEST_recovery_scrub_1 td/osd-recovery-scrub 2024-09-18T14:23:50.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:38: TEST_recovery_scrub_1: local dir=td/osd-recovery-scrub 2024-09-18T14:23:50.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub_1: local poolname=test 2024-09-18T14:23:50.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:41: TEST_recovery_scrub_1: TESTDATA=testdata.31256 2024-09-18T14:23:50.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub_1: OSDS=4 2024-09-18T14:23:50.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:43: TEST_recovery_scrub_1: PGS=1 2024-09-18T14:23:50.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub_1: OBJECTS=100 2024-09-18T14:23:50.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub_1: ERRORS=0 2024-09-18T14:23:50.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:47: TEST_recovery_scrub_1: setup td/osd-recovery-scrub 2024-09-18T14:23:50.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-09-18T14:23:50.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-09-18T14:23:50.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-09-18T14:23:50.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:23:50.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-09-18T14:23:50.083 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:23:50.083 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:23:50.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:23:50.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:23:50.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:23:50.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:23:50.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:23:50.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:23:50.087 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:23:50.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:23:50.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:23:50.090 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:23:50.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:23:50.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:23:50.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:23:50.092 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:23:50.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:23:50.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:23:50.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-09-18T14:23:50.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:23:50.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:23:50.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:23:50.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.31256 2024-09-18T14:23:50.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:23:50.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:23:50.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-09-18T14:23:50.101 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:23:50.101 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:23:50.101 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:23:50.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.31256 2024-09-18T14:23:50.103 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:23:50.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T14:23:50.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T14:23:50.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:23:50.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-09-18T14:23:50.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:48: TEST_recovery_scrub_1: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true --osd_scrub_interval_randomize_ratio=0.0 2024-09-18T14:23:50.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-09-18T14:23:50.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:23:50.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:23:50.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:23:50.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-09-18T14:23:50.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true --osd_scrub_interval_randomize_ratio=0.0 2024-09-18T14:23:50.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:23:50.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:23:50.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:23:50.498 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:23:50.498 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:23:50.498 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:23:50.498 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:23:50.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true --osd_scrub_interval_randomize_ratio=0.0 2024-09-18T14:23:50.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:23:50.779 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:23:50.779 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:23:50.779 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:23:50.779 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:23:50.780 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:23:50.780 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:23:50.780 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:23:50.781 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:23:50.781 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:23:50.781 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:23:50.781 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:23:50.781 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.31256/ceph-mon.a.asok 2024-09-18T14:23:50.782 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:23:50.782 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.31256/ceph-mon.a.asok config get fsid 2024-09-18T14:23:50.874 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:23:50.874 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:23:50.874 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:23:50.874 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:23:50.875 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:23:50.876 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:23:50.876 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:23:50.876 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:23:50.876 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:23:50.876 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:23:50.876 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:23:50.876 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.31256/ceph-mon.a.asok 2024-09-18T14:23:50.876 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:23:50.877 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.31256/ceph-mon.a.asok config get mon_host 2024-09-18T14:23:50.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:50: TEST_recovery_scrub_1: run_mgr td/osd-recovery-scrub x 2024-09-18T14:23:50.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-09-18T14:23:50.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:23:50.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:23:50.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:23:50.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-09-18T14:23:50.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:23:51.811 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:23:51.811 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:23:51.811 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:23:51.812 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:23:51.812 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:23:51.812 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:23:51.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:23:51.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:23:51.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:23:51.850 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: expr 4 - 1 2024-09-18T14:23:51.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: seq 0 3 2024-09-18T14:23:51.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:23:51.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=false 2024-09-18T14:23:51.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-09-18T14:23:51.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:23:51.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:23:51.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:23:51.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-09-18T14:23:51.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 ' 2024-09-18T14:23:51.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:23:51.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:23:51.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:23:51.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-09-18T14:23:51.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-09-18T14:23:51.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:23:51.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:23:51.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-18T14:23:51.860 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:23:51.860 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:23:51.860 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:23:51.860 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:23:51.860 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:23:51.860 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:23:51.860 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:23:51.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:23:51.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:23:51.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:23:51.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:23:51.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-18T14:23:51.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-18T14:23:51.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:23:51.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:23:51.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:23:51.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:23:51.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:23:51.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-09-18T14:23:51.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-09-18T14:23:51.861 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:23:51.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cc0406e7-9b73-464c-8cfc-ac0cd9eef285 2024-09-18T14:23:51.862 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 cc0406e7-9b73-464c-8cfc-ac0cd9eef285 2024-09-18T14:23:51.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 cc0406e7-9b73-464c-8cfc-ac0cd9eef285' 2024-09-18T14:23:51.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:23:51.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB34upmDtkdNBAAoXdGNXMQAsvuvV4Zgvxwjg== 2024-09-18T14:23:51.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB34upmDtkdNBAAoXdGNXMQAsvuvV4Zgvxwjg=="}' 2024-09-18T14:23:51.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cc0406e7-9b73-464c-8cfc-ac0cd9eef285 -i td/osd-recovery-scrub/0/new.json 2024-09-18T14:23:52.154 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:23:52.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-09-18T14:23:52.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQB34upmDtkdNBAAoXdGNXMQAsvuvV4Zgvxwjg== --osd-uuid cc0406e7-9b73-464c-8cfc-ac0cd9eef285 2024-09-18T14:23:52.193 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:23:52.188+0000 7fc8cf734540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:23:52.196 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:23:52.192+0000 7fc8cf734540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:23:52.198 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:23:52.192+0000 7fc8cf734540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:23:52.199 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:23:52.192+0000 7fc8cf734540 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-09-18T14:23:52.199 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:23:52.192+0000 7fc8cf734540 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-09-18T14:23:52.199 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:23:52.192+0000 7fc8cf734540 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-09-18T14:23:52.245 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:23:52.240+0000 7fc8cf734540 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-09-18T14:23:58.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-09-18T14:23:58.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:23:58.323 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T14:23:58.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:23:58.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:23:58.716 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T14:23:58.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:23:58.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:23:58.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-09-18T14:23:58.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:23:58.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:23:58.745 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:23:58.740+0000 7f6c580ef540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:23:58.749 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:23:58.744+0000 7f6c580ef540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:23:58.752 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:23:58.744+0000 7f6c580ef540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:23:58.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:23:58.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:23:58.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:23:58.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:23:58.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:23:58.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:58.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:23:58.998 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:23:58.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:58.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:23:59.299 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:23:59.292+0000 7f6c580ef540 -1 Falling back to public interface 2024-09-18T14:23:59.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:00.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:00.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:00.411 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:24:00.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:24:00.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:00.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:24:00.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:01.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:01.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:01.723 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:24:01.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:24:01.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:01.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:24:02.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:02.042 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:24:02.036+0000 7f6c580ef540 -1 osd.0 0 log_to_monitors true 2024-09-18T14:24:03.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:03.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:03.024 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:24:03.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:24:03.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:03.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:24:03.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:04.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:04.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:04.338 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:24:04.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:24:04.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:04.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:24:04.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:05.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:05.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:05.778 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:24:05.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:24:05.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:05.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:24:06.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:07.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:07.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:07.085 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:24:07.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:24:07.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:07.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:24:07.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:08.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:08.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:08.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:24:08.401 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:24:08.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:08.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:24:08.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:09.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:09.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:09.840 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T14:24:09.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:24:09.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:09.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:24:10.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:11.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:11.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:11.155 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T14:24:11.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:24:11.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:11.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:24:11.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:12.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:12.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:12.470 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T14:24:12.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:24:12.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:12.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:24:12.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:13.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:13.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:13.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:24:13.853 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T14:24:13.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:13.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:24:14.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:15.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:15.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:15.324 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T14:24:15.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:24:15.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:15.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:24:15.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:16.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:16.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:16.905 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T14:24:16.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:24:16.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:16.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:24:17.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:18.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:18.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:18.332 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T14:24:18.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:24:18.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:18.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:24:19.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:20.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:20.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:20.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:24:20.109 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T14:24:20.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:20.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:24:20.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:21.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:21.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:21.557 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T14:24:21.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:24:21.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:21.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:24:22.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:23.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:23.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:23.117 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T14:24:23.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:24:23.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:23.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:24:23.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:24.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:24.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:24.650 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T14:24:24.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:24:24.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:24.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:24:25.014 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/32376,v1:127.0.0.1:6803/32376] [v2:127.0.0.1:6804/32376,v1:127.0.0.1:6805/32376] exists,up cc0406e7-9b73-464c-8cfc-ac0cd9eef285 2024-09-18T14:24:25.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:24:25.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:24:25.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:24:25.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:24:25.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=false 2024-09-18T14:24:25.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-09-18T14:24:25.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:24:25.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T14:24:25.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:24:25.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-09-18T14:24:25.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 ' 2024-09-18T14:24:25.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:24:25.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:24:25.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:24:25.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-09-18T14:24:25.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-09-18T14:24:25.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:24:25.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:24:25.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-18T14:24:25.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:24:25.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:24:25.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:24:25.019 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:24:25.019 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:24:25.019 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:24:25.019 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:24:25.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:24:25.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:24:25.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:24:25.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:24:25.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-18T14:24:25.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-18T14:24:25.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:24:25.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:24:25.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:24:25.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:24:25.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:24:25.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-09-18T14:24:25.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-09-18T14:24:25.020 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:24:25.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c6065bea-29db-45e3-bd0e-0bf636edbf8b 2024-09-18T14:24:25.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c6065bea-29db-45e3-bd0e-0bf636edbf8b' 2024-09-18T14:24:25.022 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 c6065bea-29db-45e3-bd0e-0bf636edbf8b 2024-09-18T14:24:25.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:24:25.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCZ4upmGPppAhAAcF7sJxjzGvIX+InJq9AZYA== 2024-09-18T14:24:25.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCZ4upmGPppAhAAcF7sJxjzGvIX+InJq9AZYA=="}' 2024-09-18T14:24:25.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c6065bea-29db-45e3-bd0e-0bf636edbf8b -i td/osd-recovery-scrub/1/new.json 2024-09-18T14:24:25.562 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:24:25.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-09-18T14:24:25.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCZ4upmGPppAhAAcF7sJxjzGvIX+InJq9AZYA== --osd-uuid c6065bea-29db-45e3-bd0e-0bf636edbf8b 2024-09-18T14:24:25.615 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:24:25.611+0000 7fbcf8921540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:24:25.620 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:24:25.619+0000 7fbcf8921540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:24:25.622 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:24:25.619+0000 7fbcf8921540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:24:25.624 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:24:25.619+0000 7fbcf8921540 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-09-18T14:24:25.624 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:24:25.619+0000 7fbcf8921540 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-09-18T14:24:25.624 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:24:25.619+0000 7fbcf8921540 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-09-18T14:24:25.691 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:24:25.687+0000 7fbcf8921540 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-09-18T14:24:31.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-09-18T14:24:31.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:24:31.587 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T14:24:31.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T14:24:31.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:24:31.975 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T14:24:31.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T14:24:31.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:24:31.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:24:31.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:24:31.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-09-18T14:24:32.002 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:24:31.999+0000 7f8807bd6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:24:32.018 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:24:32.015+0000 7f8807bd6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:24:32.020 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:24:32.015+0000 7f8807bd6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:24:32.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T14:24:32.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:24:32.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:24:32.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:24:32.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:24:32.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:32.270 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:24:32.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:24:32.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:32.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:24:32.559 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:24:32.555+0000 7f8807bd6540 -1 Falling back to public interface 2024-09-18T14:24:32.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:33.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:33.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:33.609 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:24:33.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:24:33.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:33.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:24:33.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:34.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:34.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:34.913 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:24:34.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:24:34.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:34.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:24:35.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:35.365 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:24:35.363+0000 7f8807bd6540 -1 osd.1 0 log_to_monitors true 2024-09-18T14:24:36.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:36.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:36.226 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:24:36.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:24:36.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:36.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:24:36.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:37.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:37.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:37.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:24:37.622 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:24:37.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:37.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:24:37.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:38.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:38.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:38.957 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:24:38.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:24:38.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:38.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:24:39.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:40.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:40.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:40.398 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:24:40.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:24:40.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:40.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:24:40.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:41.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:41.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:41.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:24:41.710 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:24:41.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:41.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:24:42.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:43.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:43.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:43.162 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T14:24:43.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:24:43.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:43.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:24:43.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:44.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:44.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:44.562 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T14:24:44.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:24:44.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:44.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:24:44.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:45.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:45.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:45.875 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T14:24:45.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:24:45.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:45.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:24:46.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:47.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:47.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:47.206 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T14:24:47.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:24:47.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:47.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:24:47.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:48.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:48.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:48.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:24:48.790 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T14:24:48.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:48.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:24:49.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:50.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:50.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:50.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:24:50.233 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T14:24:50.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:50.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:24:50.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:51.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:51.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:51.640 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T14:24:51.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:24:51.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:51.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:24:52.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:53.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:53.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:53.053 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T14:24:53.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:24:53.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:53.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:24:53.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:54.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:54.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:54.704 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T14:24:54.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:24:54.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:54.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:24:55.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:56.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:56.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:56.176 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T14:24:56.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:24:56.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:56.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:24:56.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:57.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:57.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:57.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:24:57.728 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T14:24:57.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:57.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:24:58.051 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/34155,v1:127.0.0.1:6811/34155] [v2:127.0.0.1:6812/34155,v1:127.0.0.1:6813/34155] exists,up c6065bea-29db-45e3-bd0e-0bf636edbf8b 2024-09-18T14:24:58.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:24:58.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:24:58.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:24:58.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:24:58.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=false 2024-09-18T14:24:58.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-09-18T14:24:58.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:24:58.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T14:24:58.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:24:58.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-09-18T14:24:58.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 ' 2024-09-18T14:24:58.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:24:58.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:24:58.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:24:58.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-09-18T14:24:58.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-09-18T14:24:58.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:24:58.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:24:58.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-18T14:24:58.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:24:58.055 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:24:58.055 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:24:58.058 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:24:58.058 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:24:58.059 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:24:58.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:24:58.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:24:58.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:24:58.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:24:58.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:24:58.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-18T14:24:58.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-18T14:24:58.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:24:58.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:24:58.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:24:58.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:24:58.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:24:58.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-09-18T14:24:58.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-09-18T14:24:58.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:24:58.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a9c5e508-eba1-4fd3-b14d-fe85d0fe5eee 2024-09-18T14:24:58.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 a9c5e508-eba1-4fd3-b14d-fe85d0fe5eee' 2024-09-18T14:24:58.061 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 a9c5e508-eba1-4fd3-b14d-fe85d0fe5eee 2024-09-18T14:24:58.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:24:58.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC64upmRtG8BBAA4D/FT6pyBUllnk/Mk2o4gg== 2024-09-18T14:24:58.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC64upmRtG8BBAA4D/FT6pyBUllnk/Mk2o4gg=="}' 2024-09-18T14:24:58.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a9c5e508-eba1-4fd3-b14d-fe85d0fe5eee -i td/osd-recovery-scrub/2/new.json 2024-09-18T14:24:58.459 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:24:58.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-09-18T14:24:58.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQC64upmRtG8BBAA4D/FT6pyBUllnk/Mk2o4gg== --osd-uuid a9c5e508-eba1-4fd3-b14d-fe85d0fe5eee 2024-09-18T14:24:58.508 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:24:58.503+0000 7f27dd3ab540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:24:58.517 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:24:58.515+0000 7f27dd3ab540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:24:58.518 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:24:58.515+0000 7f27dd3ab540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:24:58.519 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:24:58.515+0000 7f27dd3ab540 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-09-18T14:24:58.519 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:24:58.515+0000 7f27dd3ab540 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-09-18T14:24:58.519 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:24:58.515+0000 7f27dd3ab540 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-09-18T14:24:58.596 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:24:58.595+0000 7f27dd3ab540 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-09-18T14:25:04.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-09-18T14:25:04.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:25:04.917 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T14:25:04.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T14:25:04.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:25:05.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T14:25:05.307 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T14:25:05.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:25:05.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:25:05.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:25:05.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-09-18T14:25:05.335 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:25:05.331+0000 7f15fb9a9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:25:05.338 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:25:05.335+0000 7f15fb9a9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:25:05.344 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:25:05.335+0000 7f15fb9a9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:25:05.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T14:25:05.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:25:05.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:25:05.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:25:05.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:25:05.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:05.607 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:25:05.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:25:05.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:05.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:25:05.883 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:25:05.879+0000 7f15fb9a9540 -1 Falling back to public interface 2024-09-18T14:25:05.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:06.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:06.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:06.957 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:25:06.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:25:06.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:06.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:25:07.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:08.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:08.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:08.269 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:25:08.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:25:08.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:08.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:25:08.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:08.675 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:25:08.651+0000 7f15fb9a9540 -1 osd.2 0 log_to_monitors true 2024-09-18T14:25:09.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:09.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:09.578 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:25:09.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:25:09.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:09.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:25:09.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:10.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:10.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:10.890 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:25:10.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:25:10.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:10.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:25:11.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:12.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:12.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:12.256 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:25:12.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:25:12.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:12.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:25:12.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:13.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:13.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:13.607 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:25:13.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:25:13.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:13.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:25:13.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:14.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:14.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:14.921 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:25:14.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:25:14.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:14.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:25:15.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:16.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:16.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:16.229 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T14:25:16.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:25:16.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:16.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:25:16.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:17.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:17.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:17.585 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T14:25:17.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:25:17.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:17.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:25:17.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:18.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:18.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:18.904 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T14:25:18.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:25:18.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:18.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:25:19.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:20.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:20.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:20.266 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T14:25:20.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:25:20.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:20.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:25:20.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:21.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:21.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:21.683 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T14:25:21.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:25:21.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:21.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:25:22.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:23.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:23.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:23.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:25:23.329 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T14:25:23.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:23.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:25:23.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:24.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:24.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:24.745 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T14:25:24.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:25:24.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:24.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:25:25.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:26.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:26.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:26.249 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T14:25:26.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:25:26.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:26.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:25:27.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:28.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:28.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:28.050 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T14:25:28.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:25:28.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:28.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:25:28.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:29.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:29.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:29.844 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T14:25:29.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:25:29.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:29.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:25:30.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:31.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:31.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:31.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:25:31.424 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T14:25:31.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:31.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:25:31.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:32.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:32.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:32.845 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T14:25:32.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T14:25:32.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:32.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:25:33.177 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/35946,v1:127.0.0.1:6819/35946] [v2:127.0.0.1:6820/35946,v1:127.0.0.1:6821/35946] exists,up a9c5e508-eba1-4fd3-b14d-fe85d0fe5eee 2024-09-18T14:25:33.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:25:33.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:25:33.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:25:33.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:25:33.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=false 2024-09-18T14:25:33.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-09-18T14:25:33.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:25:33.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T14:25:33.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:25:33.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-09-18T14:25:33.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 ' 2024-09-18T14:25:33.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:25:33.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:25:33.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:25:33.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-09-18T14:25:33.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-09-18T14:25:33.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:25:33.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:25:33.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-18T14:25:33.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:25:33.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:25:33.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:25:33.183 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:25:33.183 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:25:33.183 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:25:33.183 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:25:33.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:25:33.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:25:33.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:25:33.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:25:33.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-18T14:25:33.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-18T14:25:33.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:25:33.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:25:33.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:25:33.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:25:33.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:25:33.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-09-18T14:25:33.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-09-18T14:25:33.185 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:25:33.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8d0b8d42-c3c7-4cf4-afb8-fc95d22f9819 2024-09-18T14:25:33.186 INFO:tasks.workunit.client.0.smithi046.stdout:add osd3 8d0b8d42-c3c7-4cf4-afb8-fc95d22f9819 2024-09-18T14:25:33.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 8d0b8d42-c3c7-4cf4-afb8-fc95d22f9819' 2024-09-18T14:25:33.187 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:25:33.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDd4upmsd5QDBAAUD1/kXoFaaCcrNEwNWv+4w== 2024-09-18T14:25:33.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDd4upmsd5QDBAAUD1/kXoFaaCcrNEwNWv+4w=="}' 2024-09-18T14:25:33.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8d0b8d42-c3c7-4cf4-afb8-fc95d22f9819 -i td/osd-recovery-scrub/3/new.json 2024-09-18T14:25:33.713 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:25:33.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-09-18T14:25:33.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQDd4upmsd5QDBAAUD1/kXoFaaCcrNEwNWv+4w== --osd-uuid 8d0b8d42-c3c7-4cf4-afb8-fc95d22f9819 2024-09-18T14:25:33.763 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:25:33.759+0000 7ffbd3b4b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:25:33.767 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:25:33.763+0000 7ffbd3b4b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:25:33.769 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:25:33.767+0000 7ffbd3b4b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:25:33.769 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:25:33.767+0000 7ffbd3b4b540 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-09-18T14:25:33.769 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:25:33.767+0000 7ffbd3b4b540 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-09-18T14:25:33.769 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:25:33.767+0000 7ffbd3b4b540 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-09-18T14:25:33.819 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:25:33.815+0000 7ffbd3b4b540 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-09-18T14:25:40.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-09-18T14:25:40.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:25:40.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T14:25:40.009 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd3 key to auth repository 2024-09-18T14:25:40.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:25:40.399 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.3 2024-09-18T14:25:40.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T14:25:40.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-09-18T14:25:40.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:25:40.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:25:40.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:25:40.429 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:25:40.426+0000 7f4126a08540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:25:40.432 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:25:40.430+0000 7f4126a08540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:25:40.433 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:25:40.430+0000 7f4126a08540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:25:40.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T14:25:40.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:25:40.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T14:25:40.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:25:40.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:25:40.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:40.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:25:40.748 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:25:40.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:40.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:25:40.974 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:25:40.970+0000 7f4126a08540 -1 Falling back to public interface 2024-09-18T14:25:41.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:42.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:42.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:42.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:25:42.094 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:25:42.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:42.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:25:42.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:43.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:43.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:43.406 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:25:43.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:25:43.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:43.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:25:43.686 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:25:43.682+0000 7f4126a08540 -1 osd.3 0 log_to_monitors true 2024-09-18T14:25:43.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:44.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:44.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:44.718 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:25:44.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:25:44.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:44.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:25:45.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:46.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:46.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:46.035 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:25:46.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:25:46.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:25:46.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:46.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:47.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:47.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:47.522 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:25:47.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:25:47.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:47.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:25:47.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:48.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:48.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:48.839 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:25:48.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:25:48.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:48.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:25:49.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:50.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:50.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:50.159 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:25:50.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:25:50.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:50.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:25:50.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:51.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:51.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:51.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:25:51.496 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T14:25:51.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:51.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:25:51.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:52.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:52.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:52.811 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T14:25:52.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:25:52.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:52.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:25:53.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:54.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:54.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:54.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:25:54.122 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T14:25:54.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:54.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:25:54.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:55.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:55.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:55.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:25:55.580 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T14:25:55.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:55.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:25:56.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:57.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:57.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:57.175 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T14:25:57.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:25:57.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:57.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:25:57.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:58.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:58.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:58.621 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T14:25:58.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:25:58.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:58.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:25:59.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:26:00.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:26:00.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:26:00.354 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T14:26:00.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:26:00.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:26:00.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:26:00.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:26:01.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:26:01.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:26:01.713 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T14:26:01.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:26:01.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:26:01.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:26:02.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:26:03.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:26:03.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:26:03.534 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T14:26:03.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:26:03.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:26:03.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:26:03.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:26:04.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:26:04.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:26:04.961 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T14:26:04.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:26:04.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:26:04.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:26:05.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:26:06.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:26:06.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:26:06.421 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T14:26:06.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:26:06.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:26:06.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:26:06.777 INFO:tasks.workunit.client.0.smithi046.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/37752,v1:127.0.0.1:6827/37752] [v2:127.0.0.1:6828/37752,v1:127.0.0.1:6829/37752] exists,up 8d0b8d42-c3c7-4cf4-afb8-fc95d22f9819 2024-09-18T14:26:06.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:26:06.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:26:06.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:26:06.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: create_pool test 1 1 2024-09-18T14:26:06.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-18T14:26:07.225 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'test' created 2024-09-18T14:26:07.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:26:08.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:58: TEST_recovery_scrub_1: wait_for_clean 2024-09-18T14:26:08.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:26:08.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:26:08.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:26:08.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T14:26:08.239 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:26:08.240 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:26:08.240 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:26:08.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:26:08.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:26:08.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:26:08.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:26:08.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:26:08.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:26:08.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:26:08.447 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:26:08.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:26:08.754 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:26:08.755 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T14:26:08.755 INFO:tasks.workunit.client.0.smithi046.stderr:3' 2024-09-18T14:26:08.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:26:08.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:26:08.755 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:26:08.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836506 2024-09-18T14:26:08.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836506 2024-09-18T14:26:08.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836506' 2024-09-18T14:26:08.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:26:08.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:26:09.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672981 2024-09-18T14:26:09.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672981 2024-09-18T14:26:09.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836506 1-42949672981' 2024-09-18T14:26:09.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:26:09.067 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:26:09.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509456 2024-09-18T14:26:09.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509456 2024-09-18T14:26:09.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836506 1-42949672981 2-64424509456' 2024-09-18T14:26:09.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:26:09.246 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T14:26:09.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345931 2024-09-18T14:26:09.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345931 2024-09-18T14:26:09.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836506 1-42949672981 2-64424509456 3-85899345931' 2024-09-18T14:26:09.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:26:09.380 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836506 2024-09-18T14:26:09.381 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:26:09.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:26:09.384 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836506 2024-09-18T14:26:09.384 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:26:09.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836506 2024-09-18T14:26:09.387 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836506 2024-09-18T14:26:09.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-18T14:26:09.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:26:09.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836506 -lt 21474836506 2024-09-18T14:26:09.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:26:09.685 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672981 2024-09-18T14:26:09.686 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:26:09.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:26:09.689 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672981 2024-09-18T14:26:09.689 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:26:09.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672981 2024-09-18T14:26:09.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-09-18T14:26:09.692 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672981 2024-09-18T14:26:09.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:26:09.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672981 -lt 42949672981 2024-09-18T14:26:09.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:26:09.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509456 2024-09-18T14:26:09.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:26:09.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:26:09.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509456 2024-09-18T14:26:09.998 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:26:09.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509456 2024-09-18T14:26:09.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-09-18T14:26:09.999 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509456 2024-09-18T14:26:10.000 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:26:10.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509455 -lt 64424509456 2024-09-18T14:26:10.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:26:11.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:26:11.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:26:11.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509456 -lt 64424509456 2024-09-18T14:26:11.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:26:11.687 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345931 2024-09-18T14:26:11.687 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:26:11.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T14:26:11.690 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345931 2024-09-18T14:26:11.690 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:26:11.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345931 2024-09-18T14:26:11.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2024-09-18T14:26:11.692 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.3 seq 85899345931 2024-09-18T14:26:11.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T14:26:12.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345931 -lt 85899345931 2024-09-18T14:26:12.044 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:26:12.045 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:26:12.045 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:26:12.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T14:26:12.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:26:12.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:26:12.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:26:12.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:26:12.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:26:12.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:26:12.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:26:12.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T14:26:12.826 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:26:12.827 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:26:12.827 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:26:13.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T14:26:13.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:26:13.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:26:13.212 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:59: TEST_recovery_scrub_1: ceph osd dump 2024-09-18T14:26:13.212 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:59: TEST_recovery_scrub_1: grep '^pool.*['\'']test['\'']' 2024-09-18T14:26:13.212 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:59: TEST_recovery_scrub_1: awk '{ print $2 }' 2024-09-18T14:26:13.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:59: TEST_recovery_scrub_1: poolid=1 2024-09-18T14:26:13.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:61: TEST_recovery_scrub_1: ceph pg dump pgs 2024-09-18T14:26:13.893 INFO:tasks.workunit.client.0.smithi046.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:26:13.893 INFO:tasks.workunit.client.0.smithi046.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:26:07.971003+0000 0'0 22:7 [1] 1 [1] 1 0'0 2024-09-18T14:26:07.136476+0000 0'0 2024-09-18T14:26:07.136476+0000 0 0 periodic scrub scheduled @ 2024-09-19T23:58:19.168720+0000 0 0 2024-09-18T14:26:13.893 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-09-18T14:26:13.894 INFO:tasks.workunit.client.0.smithi046.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:26:13.896 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:26:13.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: dd if=/dev/urandom of=testdata.31256 bs=1M count=50 2024-09-18T14:26:14.059 INFO:tasks.workunit.client.0.smithi046.stderr:50+0 records in 2024-09-18T14:26:14.059 INFO:tasks.workunit.client.0.smithi046.stderr:50+0 records out 2024-09-18T14:26:14.059 INFO:tasks.workunit.client.0.smithi046.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.152681 s, 343 MB/s 2024-09-18T14:26:14.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: seq 1 100 2024-09-18T14:26:14.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:14.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj1 testdata.31256 2024-09-18T14:26:15.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:15.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj2 testdata.31256 2024-09-18T14:26:17.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:17.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj3 testdata.31256 2024-09-18T14:26:18.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:18.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj4 testdata.31256 2024-09-18T14:26:20.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:20.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj5 testdata.31256 2024-09-18T14:26:21.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:21.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj6 testdata.31256 2024-09-18T14:26:22.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:22.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj7 testdata.31256 2024-09-18T14:26:24.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:24.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj8 testdata.31256 2024-09-18T14:26:25.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:25.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj9 testdata.31256 2024-09-18T14:26:27.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:27.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj10 testdata.31256 2024-09-18T14:26:29.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:29.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj11 testdata.31256 2024-09-18T14:26:30.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:30.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj12 testdata.31256 2024-09-18T14:26:32.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:32.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj13 testdata.31256 2024-09-18T14:26:34.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:34.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj14 testdata.31256 2024-09-18T14:26:35.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:35.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj15 testdata.31256 2024-09-18T14:26:37.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:37.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj16 testdata.31256 2024-09-18T14:26:39.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:39.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj17 testdata.31256 2024-09-18T14:26:40.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:40.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj18 testdata.31256 2024-09-18T14:26:42.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:42.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj19 testdata.31256 2024-09-18T14:26:44.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:44.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj20 testdata.31256 2024-09-18T14:26:45.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:45.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj21 testdata.31256 2024-09-18T14:26:47.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:47.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj22 testdata.31256 2024-09-18T14:26:49.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:49.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj23 testdata.31256 2024-09-18T14:26:51.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:51.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj24 testdata.31256 2024-09-18T14:26:52.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:52.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj25 testdata.31256 2024-09-18T14:26:54.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:54.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj26 testdata.31256 2024-09-18T14:26:55.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:55.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj27 testdata.31256 2024-09-18T14:26:57.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:57.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj28 testdata.31256 2024-09-18T14:26:59.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:26:59.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj29 testdata.31256 2024-09-18T14:27:00.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:00.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj30 testdata.31256 2024-09-18T14:27:02.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:02.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj31 testdata.31256 2024-09-18T14:27:04.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:04.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj32 testdata.31256 2024-09-18T14:27:05.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:05.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj33 testdata.31256 2024-09-18T14:27:07.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:07.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj34 testdata.31256 2024-09-18T14:27:09.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:09.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj35 testdata.31256 2024-09-18T14:27:10.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:10.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj36 testdata.31256 2024-09-18T14:27:12.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:12.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj37 testdata.31256 2024-09-18T14:27:14.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:14.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj38 testdata.31256 2024-09-18T14:27:16.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:16.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj39 testdata.31256 2024-09-18T14:27:17.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:17.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj40 testdata.31256 2024-09-18T14:27:19.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:19.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj41 testdata.31256 2024-09-18T14:27:20.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:20.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj42 testdata.31256 2024-09-18T14:27:22.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:22.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj43 testdata.31256 2024-09-18T14:27:24.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:24.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj44 testdata.31256 2024-09-18T14:27:26.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:26.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj45 testdata.31256 2024-09-18T14:27:27.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:27.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj46 testdata.31256 2024-09-18T14:27:29.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:29.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj47 testdata.31256 2024-09-18T14:27:31.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:31.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj48 testdata.31256 2024-09-18T14:27:32.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:32.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj49 testdata.31256 2024-09-18T14:27:34.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:34.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj50 testdata.31256 2024-09-18T14:27:36.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:36.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj51 testdata.31256 2024-09-18T14:27:38.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:38.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj52 testdata.31256 2024-09-18T14:27:39.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:39.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj53 testdata.31256 2024-09-18T14:27:41.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:41.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj54 testdata.31256 2024-09-18T14:27:42.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:42.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj55 testdata.31256 2024-09-18T14:27:44.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:44.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj56 testdata.31256 2024-09-18T14:27:46.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:46.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj57 testdata.31256 2024-09-18T14:27:47.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:47.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj58 testdata.31256 2024-09-18T14:27:49.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:49.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj59 testdata.31256 2024-09-18T14:27:51.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:51.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj60 testdata.31256 2024-09-18T14:27:52.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:52.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj61 testdata.31256 2024-09-18T14:27:54.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:54.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj62 testdata.31256 2024-09-18T14:27:55.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:55.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj63 testdata.31256 2024-09-18T14:27:57.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:57.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj64 testdata.31256 2024-09-18T14:27:59.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:27:59.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj65 testdata.31256 2024-09-18T14:28:00.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:00.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj66 testdata.31256 2024-09-18T14:28:02.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:02.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj67 testdata.31256 2024-09-18T14:28:03.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:03.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj68 testdata.31256 2024-09-18T14:28:05.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:05.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj69 testdata.31256 2024-09-18T14:28:07.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:07.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj70 testdata.31256 2024-09-18T14:28:08.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:08.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj71 testdata.31256 2024-09-18T14:28:10.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:10.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj72 testdata.31256 2024-09-18T14:28:12.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:12.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj73 testdata.31256 2024-09-18T14:28:13.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:13.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj74 testdata.31256 2024-09-18T14:28:15.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:15.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj75 testdata.31256 2024-09-18T14:28:17.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:17.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj76 testdata.31256 2024-09-18T14:28:18.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:18.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj77 testdata.31256 2024-09-18T14:28:20.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:20.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj78 testdata.31256 2024-09-18T14:28:22.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:22.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj79 testdata.31256 2024-09-18T14:28:23.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:23.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj80 testdata.31256 2024-09-18T14:28:25.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:25.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj81 testdata.31256 2024-09-18T14:28:27.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:27.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj82 testdata.31256 2024-09-18T14:28:28.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:28.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj83 testdata.31256 2024-09-18T14:28:30.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:30.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj84 testdata.31256 2024-09-18T14:28:32.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:32.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj85 testdata.31256 2024-09-18T14:28:34.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:34.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj86 testdata.31256 2024-09-18T14:28:35.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:35.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj87 testdata.31256 2024-09-18T14:28:37.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:37.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj88 testdata.31256 2024-09-18T14:28:38.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:38.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj89 testdata.31256 2024-09-18T14:28:40.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:40.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj90 testdata.31256 2024-09-18T14:28:42.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:42.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj91 testdata.31256 2024-09-18T14:28:43.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:43.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj92 testdata.31256 2024-09-18T14:28:45.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:45.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj93 testdata.31256 2024-09-18T14:28:47.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:47.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj94 testdata.31256 2024-09-18T14:28:49.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:49.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj95 testdata.31256 2024-09-18T14:28:50.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:50.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj96 testdata.31256 2024-09-18T14:28:52.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:52.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj97 testdata.31256 2024-09-18T14:28:54.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:54.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj98 testdata.31256 2024-09-18T14:28:55.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:55.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj99 testdata.31256 2024-09-18T14:28:57.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-18T14:28:57.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub_1: rados -p test put obj100 testdata.31256 2024-09-18T14:28:59.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: rm -f testdata.31256 2024-09-18T14:28:59.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: ceph osd pool set test size 4 2024-09-18T14:28:59.515 INFO:tasks.workunit.client.0.smithi046.stderr:set pool 1 size to 4 2024-09-18T14:28:59.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:73: TEST_recovery_scrub_1: set -o pipefail 2024-09-18T14:28:59.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub_1: count=0 2024-09-18T14:28:59.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:75: TEST_recovery_scrub_1: true 2024-09-18T14:28:59.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-09-18T14:28:59.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-09-18T14:28:59.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: grep -q true 2024-09-18T14:28:59.849 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:28:59.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: sleep 2 2024-09-18T14:29:01.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:83: TEST_recovery_scrub_1: test 0 -eq 10 2024-09-18T14:29:01.861 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:88: TEST_recovery_scrub_1: expr 0 + 1 2024-09-18T14:29:01.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:88: TEST_recovery_scrub_1: count=1 2024-09-18T14:29:01.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:75: TEST_recovery_scrub_1: true 2024-09-18T14:29:01.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-09-18T14:29:01.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-09-18T14:29:01.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: grep -q true 2024-09-18T14:29:02.193 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:29:02.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: sleep 2 2024-09-18T14:29:04.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:83: TEST_recovery_scrub_1: test 1 -eq 10 2024-09-18T14:29:04.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:88: TEST_recovery_scrub_1: expr 1 + 1 2024-09-18T14:29:04.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:88: TEST_recovery_scrub_1: count=2 2024-09-18T14:29:04.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:75: TEST_recovery_scrub_1: true 2024-09-18T14:29:04.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-09-18T14:29:04.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-09-18T14:29:04.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: grep -q true 2024-09-18T14:29:04.831 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:29:04.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: sleep 2 2024-09-18T14:29:06.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:83: TEST_recovery_scrub_1: test 2 -eq 10 2024-09-18T14:29:06.843 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:88: TEST_recovery_scrub_1: expr 2 + 1 2024-09-18T14:29:06.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:88: TEST_recovery_scrub_1: count=3 2024-09-18T14:29:06.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:75: TEST_recovery_scrub_1: true 2024-09-18T14:29:06.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-09-18T14:29:06.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-09-18T14:29:06.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: grep -q true 2024-09-18T14:29:07.134 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:29:07.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:80: TEST_recovery_scrub_1: break 2024-09-18T14:29:07.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:90: TEST_recovery_scrub_1: set +o pipefail 2024-09-18T14:29:07.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:91: TEST_recovery_scrub_1: ceph pg dump pgs 2024-09-18T14:29:07.449 INFO:tasks.workunit.client.0.smithi046.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:29:07.449 INFO:tasks.workunit.client.0.smithi046.stdout:1.0 100 0 300 0 0 5242880000 0 0 1300 0 1300 active+recovering+undersized+degraded+remapped 2024-09-18T14:29:03.735513+0000 23'1300 26:1323 [1,0,2,3] 1 [1,0] 1 0'0 2024-09-18T14:26:07.136476+0000 0'0 2024-09-18T14:26:07.136476+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:11:48.471945+0000 0 0 2024-09-18T14:29:07.449 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-09-18T14:29:07.449 INFO:tasks.workunit.client.0.smithi046.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:29:07.453 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:29:07.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:93: TEST_recovery_scrub_1: sleep 10 2024-09-18T14:29:17.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub_1: kill_daemons td/osd-recovery-scrub 2024-09-18T14:29:17.464 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:29:17.465 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:29:17.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:29:17.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:29:17.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:29:22.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:29:22.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub_1: declare -a err_strings 2024-09-18T14:29:22.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:98: TEST_recovery_scrub_1: err_strings[0]='not scheduling scrubs due to active recovery' 2024-09-18T14:29:22.792 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:100: TEST_recovery_scrub_1: expr 4 - 1 2024-09-18T14:29:22.793 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:100: TEST_recovery_scrub_1: seq 0 3 2024-09-18T14:29:22.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:100: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:29:22.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.0.log 2024-09-18T14:29:22.801 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:06.702+0000 7f6c4f649700 15 osd.0 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.801 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:09.714+0000 7f6c4f649700 15 osd.0 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.801 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:10.674+0000 7f6c4f649700 15 osd.0 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.801 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:12.654+0000 7f6c4f649700 15 osd.0 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.801 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:13.634+0000 7f6c4f649700 15 osd.0 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.801 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:16.690+0000 7f6c4f649700 15 osd.0 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.801 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:18.726+0000 7f6c4f649700 15 osd.0 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.801 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:19.774+0000 7f6c4f649700 15 osd.0 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.801 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:21.798+0000 7f6c4f649700 15 osd.0 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:100: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:29:22.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.1.log 2024-09-18T14:29:22.820 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:04.222+0000 7f87ff130700 15 osd.1 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.820 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:05.262+0000 7f87ff130700 15 osd.1 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.820 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:11.198+0000 7f87ff130700 15 osd.1 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.820 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:12.218+0000 7f87ff130700 15 osd.1 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.820 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:13.230+0000 7f87ff130700 15 osd.1 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:100: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:29:22.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.2.log 2024-09-18T14:29:22.826 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:04.550+0000 7f15f2f03700 15 osd.2 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.826 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:05.530+0000 7f15f2f03700 15 osd.2 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.826 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:06.514+0000 7f15f2f03700 15 osd.2 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.826 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:08.506+0000 7f15f2f03700 15 osd.2 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.826 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:09.530+0000 7f15f2f03700 15 osd.2 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.827 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:11.586+0000 7f15f2f03700 15 osd.2 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.827 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:17.562+0000 7f15f2f03700 15 osd.2 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.827 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:19.610+0000 7f15f2f03700 15 osd.2 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:100: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:29:22.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.3.log 2024-09-18T14:29:22.829 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:04.322+0000 7f411df62700 15 osd.3 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.829 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:10.214+0000 7f411df62700 15 osd.3 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.829 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:13.210+0000 7f411df62700 15 osd.3 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.830 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:18.174+0000 7f411df62700 15 osd.3 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.830 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:20.182+0000 7f411df62700 15 osd.3 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.830 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:29:22.174+0000 7f411df62700 15 osd.3 26 sched_scrub: not scheduling scrubs due to active recovery 2024-09-18T14:29:22.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for err_string in "${err_strings[@]}" 2024-09-18T14:29:22.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: found=false 2024-09-18T14:29:22.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:107: TEST_recovery_scrub_1: count=0 2024-09-18T14:29:22.831 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: expr 4 - 1 2024-09-18T14:29:22.832 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: seq 0 3 2024-09-18T14:29:22.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:29:22.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: grep -q 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.0.log 2024-09-18T14:29:22.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: found=true 2024-09-18T14:29:22.839 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: expr 0 + 1 2024-09-18T14:29:22.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: count=1 2024-09-18T14:29:22.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:29:22.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: grep -q 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.1.log 2024-09-18T14:29:22.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: found=true 2024-09-18T14:29:22.855 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: expr 1 + 1 2024-09-18T14:29:22.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: count=2 2024-09-18T14:29:22.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:29:22.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: grep -q 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.2.log 2024-09-18T14:29:22.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: found=true 2024-09-18T14:29:22.860 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: expr 2 + 1 2024-09-18T14:29:22.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: count=3 2024-09-18T14:29:22.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:29:22.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: grep -q 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.3.log 2024-09-18T14:29:22.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: found=true 2024-09-18T14:29:22.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: expr 3 + 1 2024-09-18T14:29:22.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:113: TEST_recovery_scrub_1: count=4 2024-09-18T14:29:22.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: '[' true = false ']' 2024-09-18T14:29:22.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:120: TEST_recovery_scrub_1: '[' 4 -eq 4 ']' 2024-09-18T14:29:22.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:123: TEST_recovery_scrub_1: teardown td/osd-recovery-scrub 2024-09-18T14:29:22.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-09-18T14:29:22.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:29:22.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-09-18T14:29:22.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:29:22.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:29:22.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:29:22.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:29:22.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:29:22.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:29:22.883 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:29:22.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:29:22.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:29:22.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:29:22.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:29:22.886 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:29:22.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:29:22.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:29:22.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:29:22.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:29:22.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:29:22.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:29:22.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-09-18T14:29:22.924 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:29:22.924 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:29:22.925 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:29:22.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.31256 2024-09-18T14:29:22.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:29:22.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:29:22.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:125: TEST_recovery_scrub_1: '[' 0 '!=' 0 ']' 2024-09-18T14:29:22.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:131: TEST_recovery_scrub_1: echo 'TEST PASSED' 2024-09-18T14:29:22.927 INFO:tasks.workunit.client.0.smithi046.stdout:TEST PASSED 2024-09-18T14:29:22.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:132: TEST_recovery_scrub_1: return 0 2024-09-18T14:29:22.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: for func in $funcs 2024-09-18T14:29:22.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: TEST_recovery_scrub_2 td/osd-recovery-scrub 2024-09-18T14:29:22.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:230: TEST_recovery_scrub_2: local dir=td/osd-recovery-scrub 2024-09-18T14:29:22.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:231: TEST_recovery_scrub_2: local poolname=test 2024-09-18T14:29:22.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:233: TEST_recovery_scrub_2: TESTDATA=testdata.31256 2024-09-18T14:29:22.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:234: TEST_recovery_scrub_2: OSDS=8 2024-09-18T14:29:22.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:235: TEST_recovery_scrub_2: PGS=32 2024-09-18T14:29:22.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:236: TEST_recovery_scrub_2: OBJECTS=40 2024-09-18T14:29:22.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:238: TEST_recovery_scrub_2: setup td/osd-recovery-scrub 2024-09-18T14:29:22.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-09-18T14:29:22.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-09-18T14:29:22.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-09-18T14:29:22.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:29:22.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-09-18T14:29:22.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:29:22.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:29:22.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:29:22.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:29:22.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:29:22.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:29:22.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:29:22.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:29:22.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:29:22.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:29:22.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:29:22.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:29:22.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:29:22.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:29:22.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:29:22.938 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:29:22.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:29:22.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:29:22.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-09-18T14:29:22.944 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:29:22.944 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:29:22.945 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:29:22.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.31256 2024-09-18T14:29:22.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:29:22.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:29:22.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-09-18T14:29:22.948 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:29:22.948 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:29:22.949 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:29:22.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.31256 2024-09-18T14:29:22.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:29:22.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T14:29:22.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:29:22.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-09-18T14:29:22.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:239: TEST_recovery_scrub_2: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true --osd_scrub_interval_randomize_ratio=0.0 2024-09-18T14:29:22.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-09-18T14:29:22.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:29:22.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:29:22.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:29:22.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-09-18T14:29:22.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true --osd_scrub_interval_randomize_ratio=0.0 2024-09-18T14:29:23.301 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:29:23.301 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:29:23.301 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:29:23.302 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:29:23.302 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:29:23.302 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:29:23.302 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:29:23.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true --osd_scrub_interval_randomize_ratio=0.0 2024-09-18T14:29:23.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:29:23.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:29:23.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:29:23.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:29:23.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:29:23.583 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:29:23.584 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:29:23.584 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:29:23.584 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:29:23.585 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:29:23.585 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:29:23.585 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:29:23.585 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.31256/ceph-mon.a.asok 2024-09-18T14:29:23.586 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:29:23.586 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.31256/ceph-mon.a.asok config get fsid 2024-09-18T14:29:23.679 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:29:23.679 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:29:23.679 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:29:23.679 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:29:23.680 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:29:23.680 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:29:23.680 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:29:23.680 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:29:23.681 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:29:23.681 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:29:23.681 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:29:23.681 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.31256/ceph-mon.a.asok 2024-09-18T14:29:23.682 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:29:23.682 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.31256/ceph-mon.a.asok config get mon_host 2024-09-18T14:29:23.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:241: TEST_recovery_scrub_2: run_mgr td/osd-recovery-scrub x 2024-09-18T14:29:23.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-09-18T14:29:23.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:29:23.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:29:23.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:29:23.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-09-18T14:29:23.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:29:24.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:29:24.625 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:29:24.625 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:29:24.625 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:29:24.625 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:29:24.626 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:29:24.626 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:29:24.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:29:24.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:29:24.660 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: expr 8 - 1 2024-09-18T14:29:24.660 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: seq 0 7 2024-09-18T14:29:24.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:29:24.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2024-09-18T14:29:24.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-09-18T14:29:24.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:29:24.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:29:24.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:29:24.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-09-18T14:29:24.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 ' 2024-09-18T14:29:24.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:29:24.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:29:24.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:29:24.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-09-18T14:29:24.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-09-18T14:29:24.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:29:24.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:29:24.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-18T14:29:24.666 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:29:24.666 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:29:24.667 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:29:24.667 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:29:24.667 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:29:24.667 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:29:24.667 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:29:24.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:29:24.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:29:24.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:29:24.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:29:24.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-18T14:29:24.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-18T14:29:24.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:29:24.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:29:24.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:29:24.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:29:24.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:29:24.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2024-09-18T14:29:24.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-09-18T14:29:24.668 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:29:24.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=72ce97b6-05fa-4105-800e-f07d1b5cb959 2024-09-18T14:29:24.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 72ce97b6-05fa-4105-800e-f07d1b5cb959' 2024-09-18T14:29:24.671 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 72ce97b6-05fa-4105-800e-f07d1b5cb959 2024-09-18T14:29:24.671 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:29:24.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDE4+pmStybKBAA5suvEjWNnP6mEoJfXa8Bpg== 2024-09-18T14:29:24.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDE4+pmStybKBAA5suvEjWNnP6mEoJfXa8Bpg=="}' 2024-09-18T14:29:24.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 72ce97b6-05fa-4105-800e-f07d1b5cb959 -i td/osd-recovery-scrub/0/new.json 2024-09-18T14:29:24.998 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:29:25.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-09-18T14:29:25.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQDE4+pmStybKBAA5suvEjWNnP6mEoJfXa8Bpg== --osd-uuid 72ce97b6-05fa-4105-800e-f07d1b5cb959 2024-09-18T14:29:25.038 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:29:25.030+0000 7faaf6c04540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:29:25.043 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:29:25.038+0000 7faaf6c04540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:29:25.044 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:29:25.038+0000 7faaf6c04540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:29:25.045 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:29:25.038+0000 7faaf6c04540 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-09-18T14:29:25.045 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:29:25.038+0000 7faaf6c04540 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-09-18T14:29:25.045 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:29:25.038+0000 7faaf6c04540 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-09-18T14:29:25.114 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:29:25.106+0000 7faaf6c04540 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-09-18T14:29:30.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-09-18T14:29:30.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:29:30.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:29:30.777 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T14:29:30.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:29:31.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:29:31.163 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T14:29:31.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2024-09-18T14:29:31.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:29:31.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:29:31.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:29:31.195 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:29:31.190+0000 7f0d52cbc540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:29:31.198 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:29:31.194+0000 7f0d52cbc540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:29:31.199 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:29:31.194+0000 7f0d52cbc540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:29:31.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:29:31.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:29:31.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:29:31.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:29:31.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:29:31.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:31.449 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:29:31.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:29:31.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:31.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:31.749 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:29:31.742+0000 7f0d52cbc540 -1 Falling back to public interface 2024-09-18T14:29:31.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:32.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:32.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:32.800 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:29:32.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:29:32.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:32.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:33.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:34.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:34.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:34.119 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:29:34.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:29:34.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:34.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:34.221 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:29:34.214+0000 7f0d52cbc540 -1 osd.0 0 log_to_monitors true 2024-09-18T14:29:34.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:35.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:35.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:35.431 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:29:35.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:29:35.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:35.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:35.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:36.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:36.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:36.786 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:29:36.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:29:36.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:36.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:37.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:38.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:38.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:38.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:29:38.186 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:29:38.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:38.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:38.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:39.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:39.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:39.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:29:39.505 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:29:39.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:39.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:39.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:40.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:40.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:40.817 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:29:40.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:29:40.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:40.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:41.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:42.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:42.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:42.167 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T14:29:42.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:29:42.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:42.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:42.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:43.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:43.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:43.481 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T14:29:43.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:29:43.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:43.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:43.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:44.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:44.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:44.790 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T14:29:44.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:29:44.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:44.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:45.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:46.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:46.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:46.192 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T14:29:46.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:29:46.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:46.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:46.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:47.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:47.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:47.510 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T14:29:47.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:29:47.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:47.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:48.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:49.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:49.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:49.039 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T14:29:49.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:29:49.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:49.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:49.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:50.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:50.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:50.765 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T14:29:50.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:29:50.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:50.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:51.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:52.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:52.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:52.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:29:52.462 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T14:29:52.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:52.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:53.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:54.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:54.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:54.191 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T14:29:54.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:29:54.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:54.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:54.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:55.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:55.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:55.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:29:55.919 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T14:29:55.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:55.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:56.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:57.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:57.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:57.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:29:57.390 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T14:29:57.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:57.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:57.712 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/42577,v1:127.0.0.1:6803/42577] [v2:127.0.0.1:6804/42577,v1:127.0.0.1:6805/42577] exists,up 72ce97b6-05fa-4105-800e-f07d1b5cb959 2024-09-18T14:29:57.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:29:57.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:29:57.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:29:57.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:29:57.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2024-09-18T14:29:57.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-09-18T14:29:57.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:29:57.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T14:29:57.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:29:57.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-09-18T14:29:57.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 ' 2024-09-18T14:29:57.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:29:57.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:29:57.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:29:57.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-09-18T14:29:57.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-09-18T14:29:57.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:29:57.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:29:57.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-18T14:29:57.715 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:29:57.715 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:29:57.715 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:29:57.716 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:29:57.716 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:29:57.716 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:29:57.717 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:29:57.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:29:57.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:29:57.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:29:57.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:29:57.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-18T14:29:57.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-18T14:29:57.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:29:57.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:29:57.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:29:57.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:29:57.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:29:57.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2024-09-18T14:29:57.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-09-18T14:29:57.718 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:29:57.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7fca2357-c7f1-4ad9-9a53-c15d8d6772a9 2024-09-18T14:29:57.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7fca2357-c7f1-4ad9-9a53-c15d8d6772a9' 2024-09-18T14:29:57.719 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 7fca2357-c7f1-4ad9-9a53-c15d8d6772a9 2024-09-18T14:29:57.719 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:29:57.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDl4+pmxqLXKxAAr9wITeICs4ENHUOeka46ZQ== 2024-09-18T14:29:57.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDl4+pmxqLXKxAAr9wITeICs4ENHUOeka46ZQ=="}' 2024-09-18T14:29:57.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7fca2357-c7f1-4ad9-9a53-c15d8d6772a9 -i td/osd-recovery-scrub/1/new.json 2024-09-18T14:29:58.121 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:29:58.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-09-18T14:29:58.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQDl4+pmxqLXKxAAr9wITeICs4ENHUOeka46ZQ== --osd-uuid 7fca2357-c7f1-4ad9-9a53-c15d8d6772a9 2024-09-18T14:29:58.168 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:29:58.161+0000 7f3a5c80f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:29:58.178 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:29:58.169+0000 7f3a5c80f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:29:58.179 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:29:58.173+0000 7f3a5c80f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:29:58.181 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:29:58.173+0000 7f3a5c80f540 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-09-18T14:29:58.181 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:29:58.173+0000 7f3a5c80f540 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-09-18T14:29:58.181 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:29:58.173+0000 7f3a5c80f540 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-09-18T14:29:58.247 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:29:58.241+0000 7f3a5c80f540 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-09-18T14:30:04.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-09-18T14:30:04.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:30:04.274 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T14:30:04.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T14:30:04.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:30:04.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T14:30:04.648 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T14:30:04.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2024-09-18T14:30:04.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:30:04.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:30:04.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:30:04.680 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:30:04.673+0000 7f3573ce0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:30:04.683 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:30:04.677+0000 7f3573ce0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:30:04.684 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:30:04.677+0000 7f3573ce0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:30:04.969 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:30:04.961+0000 7f3573ce0540 -1 Falling back to public interface 2024-09-18T14:30:05.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T14:30:05.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:30:05.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:30:05.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:30:05.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:30:05.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:05.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:30:05.019 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:30:05.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:05.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:30:05.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:06.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:06.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:06.373 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:30:06.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:30:06.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:06.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:30:06.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:07.674 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:30:07.665+0000 7f3573ce0540 -1 osd.1 0 log_to_monitors true 2024-09-18T14:30:07.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:07.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:07.686 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:30:07.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:30:07.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:07.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:30:08.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:09.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:09.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:09.108 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:30:09.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:30:09.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:09.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:30:09.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:10.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:10.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:10.422 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:30:10.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:30:10.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:10.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:30:10.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:11.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:11.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:11.730 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:30:11.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:30:11.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:11.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:30:12.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:13.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:13.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:13.215 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:30:13.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:30:13.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:13.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:30:13.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:14.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:14.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:14.425 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:30:14.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:30:14.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:14.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:30:14.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:15.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:15.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:15.733 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T14:30:15.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:30:15.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:15.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:30:16.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:17.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:17.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:17.051 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T14:30:17.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:30:17.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:17.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:30:17.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:18.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:18.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:18.432 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T14:30:18.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:30:18.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:18.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:30:18.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:19.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:19.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:19.795 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T14:30:19.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:30:19.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:19.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:30:20.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:21.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:21.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:21.108 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T14:30:21.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:30:21.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:21.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:30:21.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:22.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:22.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:22.594 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T14:30:22.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:30:22.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:22.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:30:22.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:23.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:23.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:23.952 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T14:30:23.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:30:23.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:23.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:30:24.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:25.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:25.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:25.480 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T14:30:25.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:30:25.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:25.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:30:26.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:27.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:27.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:27.405 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T14:30:27.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:30:27.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:27.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:30:28.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:29.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:29.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:29.062 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T14:30:29.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:30:29.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:29.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:30:29.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:30.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:30.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:30.459 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T14:30:30.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:30:30.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:30.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:30:31.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:32.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:32.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:32.053 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T14:30:32.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T14:30:32.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:32.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:30:32.371 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/44356,v1:127.0.0.1:6811/44356] [v2:127.0.0.1:6812/44356,v1:127.0.0.1:6813/44356] exists,up 7fca2357-c7f1-4ad9-9a53-c15d8d6772a9 2024-09-18T14:30:32.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:30:32.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:30:32.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:30:32.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:30:32.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2024-09-18T14:30:32.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-09-18T14:30:32.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:30:32.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T14:30:32.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:30:32.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-09-18T14:30:32.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 ' 2024-09-18T14:30:32.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:30:32.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:30:32.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:30:32.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-09-18T14:30:32.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-09-18T14:30:32.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:30:32.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:30:32.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-18T14:30:32.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:30:32.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:30:32.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:30:32.382 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:30:32.383 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:30:32.383 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:30:32.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:30:32.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:30:32.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:30:32.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:30:32.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:30:32.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-18T14:30:32.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-18T14:30:32.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:30:32.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:30:32.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:30:32.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:30:32.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:30:32.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2024-09-18T14:30:32.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-09-18T14:30:32.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:30:32.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=591f83c1-6188-45d0-b850-9b32ef0265e5 2024-09-18T14:30:32.385 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 591f83c1-6188-45d0-b850-9b32ef0265e5 2024-09-18T14:30:32.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 591f83c1-6188-45d0-b850-9b32ef0265e5' 2024-09-18T14:30:32.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:30:32.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAI5OpmtkqIFxAAzL1V8VvE9anMMwWQcte4MA== 2024-09-18T14:30:32.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAI5OpmtkqIFxAAzL1V8VvE9anMMwWQcte4MA=="}' 2024-09-18T14:30:32.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 591f83c1-6188-45d0-b850-9b32ef0265e5 -i td/osd-recovery-scrub/2/new.json 2024-09-18T14:30:32.797 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:30:32.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-09-18T14:30:32.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQAI5OpmtkqIFxAAzL1V8VvE9anMMwWQcte4MA== --osd-uuid 591f83c1-6188-45d0-b850-9b32ef0265e5 2024-09-18T14:30:32.851 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:30:32.841+0000 7f13d07b5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:30:32.854 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:30:32.845+0000 7f13d07b5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:30:32.856 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:30:32.849+0000 7f13d07b5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:30:32.857 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:30:32.849+0000 7f13d07b5540 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-09-18T14:30:32.857 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:30:32.849+0000 7f13d07b5540 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-09-18T14:30:32.857 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:30:32.849+0000 7f13d07b5540 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-09-18T14:30:32.953 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:30:32.945+0000 7f13d07b5540 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-09-18T14:30:39.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-09-18T14:30:39.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:30:39.038 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T14:30:39.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T14:30:39.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:30:39.418 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T14:30:39.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T14:30:39.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2024-09-18T14:30:39.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:30:39.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:30:39.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:30:39.446 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:30:39.436+0000 7f573b43e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:30:39.449 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:30:39.440+0000 7f573b43e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:30:39.450 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:30:39.440+0000 7f573b43e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:30:39.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T14:30:39.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:30:39.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:30:39.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:30:39.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:30:39.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:39.765 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:30:39.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:30:39.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:39.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:30:39.985 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:30:39.976+0000 7f573b43e540 -1 Falling back to public interface 2024-09-18T14:30:40.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:41.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:41.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:41.051 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:30:41.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:30:41.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:41.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:30:41.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:42.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:42.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:42.364 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:30:42.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:30:42.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:42.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:30:42.668 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:30:42.660+0000 7f573b43e540 -1 osd.2 0 log_to_monitors true 2024-09-18T14:30:42.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:43.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:43.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:43.683 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:30:43.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:30:43.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:43.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:30:44.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:45.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:45.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:45.019 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:30:45.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:30:45.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:45.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:30:45.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:46.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:46.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:46.346 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:30:46.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:30:46.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:46.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:30:46.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:47.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:47.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:47.710 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:30:47.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:30:47.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:47.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:30:48.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:49.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:49.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:49.058 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:30:49.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:30:49.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:49.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:30:49.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:50.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:50.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:50.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:30:50.379 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T14:30:50.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:50.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:30:50.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:51.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:51.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:51.689 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T14:30:51.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:30:51.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:51.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:30:52.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:53.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:53.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:53.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:30:53.024 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T14:30:53.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:53.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:30:53.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:54.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:54.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:54.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:30:54.333 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T14:30:54.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:54.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:30:54.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:55.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:55.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:55.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:30:55.734 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T14:30:55.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:55.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:30:56.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:57.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:57.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:57.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:30:57.050 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T14:30:57.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:57.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:30:57.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:58.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:58.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:58.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:30:58.423 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T14:30:58.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:58.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:30:58.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:59.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:59.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:59.739 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T14:30:59.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:30:59.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:59.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:31:00.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:01.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:01.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:01.287 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T14:31:01.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:31:01.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:01.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:31:01.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:02.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:02.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:02.679 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T14:31:02.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:31:02.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:02.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:31:02.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:04.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:04.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:04.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:31:04.001 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T14:31:04.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:04.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:31:04.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:05.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:05.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:05.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T14:31:05.394 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T14:31:05.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:05.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:31:05.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:06.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:06.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:06.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-18T14:31:06.697 INFO:tasks.workunit.client.0.smithi046.stdout:20 2024-09-18T14:31:06.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:06.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:31:06.999 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/46161,v1:127.0.0.1:6819/46161] [v2:127.0.0.1:6820/46161,v1:127.0.0.1:6821/46161] exists,up 591f83c1-6188-45d0-b850-9b32ef0265e5 2024-09-18T14:31:07.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:31:07.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:31:07.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:31:07.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:31:07.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2024-09-18T14:31:07.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-09-18T14:31:07.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:31:07.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T14:31:07.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:31:07.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-09-18T14:31:07.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 ' 2024-09-18T14:31:07.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:31:07.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:31:07.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:31:07.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-09-18T14:31:07.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-09-18T14:31:07.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:31:07.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:31:07.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-18T14:31:07.006 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:31:07.006 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:31:07.006 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:31:07.008 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:31:07.008 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:31:07.008 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:31:07.008 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:31:07.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:31:07.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:31:07.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:31:07.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:31:07.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-18T14:31:07.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-18T14:31:07.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:31:07.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:31:07.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:31:07.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:31:07.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:31:07.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2024-09-18T14:31:07.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-09-18T14:31:07.019 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:31:07.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0f897023-d613-4bad-89a7-81bcef9832d4 2024-09-18T14:31:07.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 0f897023-d613-4bad-89a7-81bcef9832d4' 2024-09-18T14:31:07.021 INFO:tasks.workunit.client.0.smithi046.stdout:add osd3 0f897023-d613-4bad-89a7-81bcef9832d4 2024-09-18T14:31:07.021 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:31:07.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAr5Opm67lQAhAAFeP4MtY/PzIHd8inryGvbQ== 2024-09-18T14:31:07.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAr5Opm67lQAhAAFeP4MtY/PzIHd8inryGvbQ=="}' 2024-09-18T14:31:07.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0f897023-d613-4bad-89a7-81bcef9832d4 -i td/osd-recovery-scrub/3/new.json 2024-09-18T14:31:07.477 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:31:07.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-09-18T14:31:07.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQAr5Opm67lQAhAAFeP4MtY/PzIHd8inryGvbQ== --osd-uuid 0f897023-d613-4bad-89a7-81bcef9832d4 2024-09-18T14:31:07.526 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:31:07.516+0000 7f7131d7e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:31:07.530 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:31:07.520+0000 7f7131d7e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:31:07.531 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:31:07.524+0000 7f7131d7e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:31:07.533 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:31:07.524+0000 7f7131d7e540 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-09-18T14:31:07.533 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:31:07.524+0000 7f7131d7e540 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-09-18T14:31:07.533 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:31:07.524+0000 7f7131d7e540 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-09-18T14:31:07.591 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:31:07.580+0000 7f7131d7e540 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-09-18T14:31:13.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-09-18T14:31:13.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:31:13.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T14:31:13.890 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd3 key to auth repository 2024-09-18T14:31:13.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:31:14.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T14:31:14.275 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.3 2024-09-18T14:31:14.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2024-09-18T14:31:14.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:31:14.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:31:14.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:31:14.304 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:31:14.296+0000 7f87f5064540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:31:14.307 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:31:14.300+0000 7f87f5064540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:31:14.310 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:31:14.300+0000 7f87f5064540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:31:14.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T14:31:14.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:31:14.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T14:31:14.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:31:14.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:31:14.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:14.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:31:14.627 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:31:14.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:31:14.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:14.865 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:31:14.856+0000 7f87f5064540 -1 Falling back to public interface 2024-09-18T14:31:14.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:15.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:15.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:15.960 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:31:15.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:31:15.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:15.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:31:16.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:17.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:17.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:17.253 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:31:17.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:31:17.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:17.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:31:17.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:17.643 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:31:17.624+0000 7f87f5064540 -1 osd.3 0 log_to_monitors true 2024-09-18T14:31:18.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:18.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:18.558 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:31:18.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:31:18.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:18.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:31:19.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:20.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:20.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:20.149 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:31:20.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:31:20.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:20.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:31:20.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:21.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:21.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:21.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:31:21.565 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:31:21.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:21.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:31:21.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:22.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:22.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:22.876 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:31:22.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:31:22.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:22.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:31:23.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:24.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:24.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:24.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:31:24.227 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:31:24.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:24.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:31:24.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:25.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:25.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:25.641 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T14:31:25.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:31:25.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:25.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:31:25.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:26.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:26.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:26.945 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T14:31:26.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:31:26.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:26.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:31:27.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:28.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:28.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:28.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:31:28.253 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T14:31:28.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:28.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:31:28.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:29.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:29.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:29.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:31:29.718 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T14:31:29.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:29.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:31:30.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:31.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:31.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:31.085 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T14:31:31.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:31:31.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:31.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:31:31.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:32.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:32.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:32.560 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T14:31:32.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:31:32.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:32.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:31:32.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:33.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:33.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:33.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:31:33.966 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T14:31:33.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:33.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:31:34.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:35.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:35.661 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T14:31:35.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:35.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:31:35.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:35.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:31:36.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:37.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:37.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:37.485 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T14:31:37.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:31:37.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:37.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:31:38.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:39.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:39.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:39.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:31:39.065 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T14:31:39.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:39.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:31:39.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:40.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:40.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:40.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:31:40.411 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T14:31:40.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:40.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:31:40.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:41.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:41.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:41.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T14:31:41.801 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T14:31:41.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:41.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:31:42.239 INFO:tasks.workunit.client.0.smithi046.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/47991,v1:127.0.0.1:6827/47991] [v2:127.0.0.1:6828/47991,v1:127.0.0.1:6829/47991] exists,up 0f897023-d613-4bad-89a7-81bcef9832d4 2024-09-18T14:31:42.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:31:42.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:31:42.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:31:42.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:31:42.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 4 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2024-09-18T14:31:42.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-09-18T14:31:42.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:31:42.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-18T14:31:42.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:31:42.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/4 2024-09-18T14:31:42.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 ' 2024-09-18T14:31:42.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:31:42.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:31:42.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:31:42.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/4' 2024-09-18T14:31:42.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/4/journal' 2024-09-18T14:31:42.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:31:42.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:31:42.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-18T14:31:42.243 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:31:42.243 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:31:42.243 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:31:42.244 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:31:42.244 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:31:42.244 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:31:42.244 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:31:42.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:31:42.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:31:42.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:31:42.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:31:42.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-18T14:31:42.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-18T14:31:42.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:31:42.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:31:42.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:31:42.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:31:42.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:31:42.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2024-09-18T14:31:42.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/4 2024-09-18T14:31:42.246 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:31:42.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fd5c0ebc-8380-4635-9aaf-63ec889130db 2024-09-18T14:31:42.246 INFO:tasks.workunit.client.0.smithi046.stdout:add osd4 fd5c0ebc-8380-4635-9aaf-63ec889130db 2024-09-18T14:31:42.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 fd5c0ebc-8380-4635-9aaf-63ec889130db' 2024-09-18T14:31:42.247 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:31:42.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBO5OpmOSaPDxAAPX3qhHT2b9fLnqqgZSbF0A== 2024-09-18T14:31:42.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBO5OpmOSaPDxAAPX3qhHT2b9fLnqqgZSbF0A=="}' 2024-09-18T14:31:42.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fd5c0ebc-8380-4635-9aaf-63ec889130db -i td/osd-recovery-scrub/4/new.json 2024-09-18T14:31:42.661 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:31:42.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/4/new.json 2024-09-18T14:31:42.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQBO5OpmOSaPDxAAPX3qhHT2b9fLnqqgZSbF0A== --osd-uuid fd5c0ebc-8380-4635-9aaf-63ec889130db 2024-09-18T14:31:42.712 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:31:42.707+0000 7f9c9fef6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:31:42.727 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:31:42.723+0000 7f9c9fef6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:31:42.729 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:31:42.723+0000 7f9c9fef6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:31:42.730 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:31:42.723+0000 7f9c9fef6540 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-09-18T14:31:42.730 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:31:42.723+0000 7f9c9fef6540 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-09-18T14:31:42.730 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:31:42.723+0000 7f9c9fef6540 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-09-18T14:31:42.836 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:31:42.831+0000 7f9c9fef6540 -1 bluestore(td/osd-recovery-scrub/4) _read_fsid unparsable uuid 2024-09-18T14:31:48.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/4/keyring 2024-09-18T14:31:48.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:31:48.883 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd4 key to auth repository 2024-09-18T14:31:48.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-18T14:31:48.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:31:49.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-18T14:31:49.278 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.4 2024-09-18T14:31:49.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2024-09-18T14:31:49.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:31:49.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:31:49.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:31:49.306 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:31:49.299+0000 7f4e4d030540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:31:49.309 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:31:49.303+0000 7f4e4d030540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:31:49.311 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:31:49.303+0000 7f4e4d030540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:31:49.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-18T14:31:49.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:31:49.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-18T14:31:49.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:31:49.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:31:49.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:49.643 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:31:49.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:31:49.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:49.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:31:49.853 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:31:49.847+0000 7f4e4d030540 -1 Falling back to public interface 2024-09-18T14:31:49.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:50.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:50.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:50.948 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:31:50.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:31:50.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:50.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:31:51.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:52.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:52.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:52.255 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:31:52.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:31:52.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:52.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:31:52.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:52.657 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:31:52.631+0000 7f4e4d030540 -1 osd.4 0 log_to_monitors true 2024-09-18T14:31:53.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:53.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:53.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:31:53.561 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:31:53.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:53.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:31:54.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:55.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:55.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:55.059 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:31:55.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:31:55.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:55.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:31:55.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:56.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:56.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:56.370 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:31:56.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:31:56.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:56.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:31:56.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:57.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:57.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:57.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:31:57.741 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:31:57.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:57.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:31:58.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:59.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:59.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:59.060 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:31:59.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:31:59.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:59.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:31:59.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:00.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:00.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:00.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:32:00.383 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T14:32:00.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:00.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:32:00.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:01.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:01.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:01.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:32:01.733 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T14:32:01.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:01.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:32:02.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:03.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:03.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:03.046 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T14:32:03.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:32:03.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:03.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:32:03.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:04.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:04.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:04.361 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T14:32:04.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:32:04.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:04.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:32:04.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:05.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:05.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:05.667 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T14:32:05.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:32:05.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:05.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:32:06.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:07.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:07.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:07.203 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T14:32:07.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:32:07.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:07.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:32:07.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:08.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:08.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:08.759 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T14:32:08.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:32:08.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:08.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:32:09.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:10.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:10.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:10.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:32:10.072 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T14:32:10.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:10.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:32:10.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:11.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:11.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:11.889 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T14:32:11.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:32:11.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:11.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:32:12.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:13.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:13.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:13.633 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T14:32:13.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:32:13.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:13.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:32:14.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:15.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:15.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:15.057 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T14:32:15.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:32:15.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:15.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:32:15.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:16.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:16.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:16.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T14:32:16.433 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T14:32:16.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:16.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:32:16.754 INFO:tasks.workunit.client.0.smithi046.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/49795,v1:127.0.0.1:6835/49795] [v2:127.0.0.1:6836/49795,v1:127.0.0.1:6837/49795] exists,up fd5c0ebc-8380-4635-9aaf-63ec889130db 2024-09-18T14:32:16.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:32:16.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:32:16.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:32:16.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:32:16.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 5 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2024-09-18T14:32:16.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-09-18T14:32:16.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:32:16.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-18T14:32:16.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:32:16.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/5 2024-09-18T14:32:16.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 ' 2024-09-18T14:32:16.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:32:16.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:32:16.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:32:16.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/5' 2024-09-18T14:32:16.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/5/journal' 2024-09-18T14:32:16.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:32:16.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:32:16.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-18T14:32:16.758 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:32:16.758 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:32:16.758 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:32:16.758 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:32:16.758 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:32:16.758 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:32:16.760 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:32:16.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:32:16.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:32:16.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:32:16.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:32:16.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-18T14:32:16.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-18T14:32:16.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:32:16.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:32:16.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:32:16.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:32:16.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:32:16.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2024-09-18T14:32:16.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/5 2024-09-18T14:32:16.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:32:16.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8ae58f71-951b-4fde-9e1a-429b1c58741a 2024-09-18T14:32:16.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 8ae58f71-951b-4fde-9e1a-429b1c58741a' 2024-09-18T14:32:16.762 INFO:tasks.workunit.client.0.smithi046.stdout:add osd5 8ae58f71-951b-4fde-9e1a-429b1c58741a 2024-09-18T14:32:16.762 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:32:16.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBw5Opmy0ZvLhAAfhhEMYHNqea3J5j9aAl9Ug== 2024-09-18T14:32:16.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBw5Opmy0ZvLhAAfhhEMYHNqea3J5j9aAl9Ug=="}' 2024-09-18T14:32:16.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8ae58f71-951b-4fde-9e1a-429b1c58741a -i td/osd-recovery-scrub/5/new.json 2024-09-18T14:32:17.347 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:32:17.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/5/new.json 2024-09-18T14:32:17.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQBw5Opmy0ZvLhAAfhhEMYHNqea3J5j9aAl9Ug== --osd-uuid 8ae58f71-951b-4fde-9e1a-429b1c58741a 2024-09-18T14:32:17.401 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:32:17.394+0000 7fb8ff723540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:32:17.405 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:32:17.398+0000 7fb8ff723540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:32:17.407 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:32:17.398+0000 7fb8ff723540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:32:17.408 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:32:17.402+0000 7fb8ff723540 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-09-18T14:32:17.408 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:32:17.402+0000 7fb8ff723540 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-09-18T14:32:17.408 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:32:17.402+0000 7fb8ff723540 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-09-18T14:32:17.514 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:32:17.506+0000 7fb8ff723540 -1 bluestore(td/osd-recovery-scrub/5) _read_fsid unparsable uuid 2024-09-18T14:32:24.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/5/keyring 2024-09-18T14:32:24.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:32:24.354 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd5 key to auth repository 2024-09-18T14:32:24.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-18T14:32:24.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:32:24.739 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.5 2024-09-18T14:32:24.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-18T14:32:24.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2024-09-18T14:32:24.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:32:24.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:32:24.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:32:24.769 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:32:24.762+0000 7f3f4c58f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:32:24.772 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:32:24.766+0000 7f3f4c58f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:32:24.773 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:32:24.766+0000 7f3f4c58f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:32:25.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-18T14:32:25.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:32:25.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T14:32:25.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:32:25.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:32:25.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:25.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:32:25.056 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:32:25.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:25.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T14:32:25.326 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:32:25.318+0000 7f3f4c58f540 -1 Falling back to public interface 2024-09-18T14:32:25.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:26.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:26.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:26.401 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:32:26.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:32:26.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:26.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T14:32:26.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:27.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:27.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:27.711 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:32:27.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:32:27.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:27.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T14:32:28.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:28.145 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:32:28.126+0000 7f3f4c58f540 -1 osd.5 0 log_to_monitors true 2024-09-18T14:32:29.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:29.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:29.060 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:32:29.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:32:29.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:29.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T14:32:29.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:30.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:30.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:30.468 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:32:30.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:32:30.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:30.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T14:32:30.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:31.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:31.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:31.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:32:31.821 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:32:31.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:31.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T14:32:32.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:33.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:33.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:33.188 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:32:33.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:32:33.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:33.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T14:32:33.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:34.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:34.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:34.504 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:32:34.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:32:34.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:34.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T14:32:34.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:35.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:35.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:35.821 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T14:32:35.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:32:35.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:35.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T14:32:36.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:37.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:37.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:37.134 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T14:32:37.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:32:37.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:37.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T14:32:37.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:38.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:38.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:38.446 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T14:32:38.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:32:38.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:38.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T14:32:38.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:39.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:39.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:39.838 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T14:32:39.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:32:39.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:39.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T14:32:40.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:41.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:41.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:41.243 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T14:32:41.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:32:41.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:41.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T14:32:41.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:42.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:42.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:42.713 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T14:32:42.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:32:42.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:42.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T14:32:43.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:44.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:44.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:44.025 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T14:32:44.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:32:44.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:44.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T14:32:44.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:45.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:45.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:45.690 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T14:32:45.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:32:45.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:45.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T14:32:46.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:47.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:47.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:47.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:32:47.348 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T14:32:47.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:47.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T14:32:47.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:48.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:48.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:48.919 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T14:32:48.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:32:48.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:48.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T14:32:49.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:50.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:50.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:50.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:32:50.537 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T14:32:50.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:50.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T14:32:50.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:51.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:51.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:51.957 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T14:32:51.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T14:32:51.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:51.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T14:32:52.268 INFO:tasks.workunit.client.0.smithi046.stdout:osd.5 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/51599,v1:127.0.0.1:6843/51599] [v2:127.0.0.1:6844/51599,v1:127.0.0.1:6845/51599] exists,up 8ae58f71-951b-4fde-9e1a-429b1c58741a 2024-09-18T14:32:52.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:32:52.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:32:52.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:32:52.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:32:52.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 6 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2024-09-18T14:32:52.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-09-18T14:32:52.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:32:52.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-09-18T14:32:52.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:32:52.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/6 2024-09-18T14:32:52.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 ' 2024-09-18T14:32:52.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:32:52.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:32:52.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:32:52.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/6' 2024-09-18T14:32:52.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/6/journal' 2024-09-18T14:32:52.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:32:52.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:32:52.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-18T14:32:52.272 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:32:52.272 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:32:52.272 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:32:52.274 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:32:52.274 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:32:52.274 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:32:52.274 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:32:52.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:32:52.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:32:52.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:32:52.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:32:52.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-18T14:32:52.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-18T14:32:52.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:32:52.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:32:52.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:32:52.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:32:52.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:32:52.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2024-09-18T14:32:52.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/6 2024-09-18T14:32:52.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:32:52.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=54942822-d1c1-498f-a68e-8984f0b32f6a 2024-09-18T14:32:52.277 INFO:tasks.workunit.client.0.smithi046.stdout:add osd6 54942822-d1c1-498f-a68e-8984f0b32f6a 2024-09-18T14:32:52.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 54942822-d1c1-498f-a68e-8984f0b32f6a' 2024-09-18T14:32:52.277 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:32:52.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCU5Opmgv+FERAACgDKqJXmXUUHy0lLLozGSg== 2024-09-18T14:32:52.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCU5Opmgv+FERAACgDKqJXmXUUHy0lLLozGSg=="}' 2024-09-18T14:32:52.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 54942822-d1c1-498f-a68e-8984f0b32f6a -i td/osd-recovery-scrub/6/new.json 2024-09-18T14:32:52.704 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:32:52.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/6/new.json 2024-09-18T14:32:52.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQCU5Opmgv+FERAACgDKqJXmXUUHy0lLLozGSg== --osd-uuid 54942822-d1c1-498f-a68e-8984f0b32f6a 2024-09-18T14:32:52.749 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:32:52.742+0000 7f2d051b6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:32:52.761 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:32:52.754+0000 7f2d051b6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:32:52.762 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:32:52.754+0000 7f2d051b6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:32:52.764 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:32:52.754+0000 7f2d051b6540 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-09-18T14:32:52.764 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:32:52.754+0000 7f2d051b6540 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-09-18T14:32:52.764 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:32:52.754+0000 7f2d051b6540 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-09-18T14:32:52.894 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:32:52.886+0000 7f2d051b6540 -1 bluestore(td/osd-recovery-scrub/6) _read_fsid unparsable uuid 2024-09-18T14:32:58.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/6/keyring 2024-09-18T14:32:58.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:32:58.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-09-18T14:32:58.899 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd6 key to auth repository 2024-09-18T14:32:58.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:32:59.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-09-18T14:32:59.287 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.6 2024-09-18T14:32:59.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2024-09-18T14:32:59.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:32:59.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:32:59.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:32:59.317 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:32:59.310+0000 7f05b0884540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:32:59.320 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:32:59.314+0000 7f05b0884540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:32:59.325 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:32:59.318+0000 7f05b0884540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:32:59.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-09-18T14:32:59.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:32:59.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-09-18T14:32:59.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:32:59.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:32:59.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:59.584 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:32:59.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:32:59.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:59.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T14:32:59.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:59.902 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:32:59.894+0000 7f05b0884540 -1 Falling back to public interface 2024-09-18T14:33:00.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:00.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:00.877 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:33:00.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:33:00.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:00.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T14:33:01.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:02.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:02.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:02.181 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:33:02.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:33:02.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:02.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T14:33:02.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:02.834 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:33:02.826+0000 7f05b0884540 -1 osd.6 0 log_to_monitors true 2024-09-18T14:33:03.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:03.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:03.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:33:03.490 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:33:03.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:03.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T14:33:03.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:04.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:04.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:04.947 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:33:04.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:33:04.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:04.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T14:33:05.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:06.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:06.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:06.276 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:33:06.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:33:06.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:06.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T14:33:06.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:07.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:07.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:07.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:33:07.688 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:33:07.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:07.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T14:33:07.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:09.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:09.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:09.001 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:33:09.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:33:09.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:09.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T14:33:09.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:10.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:10.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:10.346 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T14:33:10.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:33:10.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:10.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T14:33:10.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:11.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:11.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:11.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:33:11.686 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T14:33:11.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:11.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T14:33:12.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:13.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:13.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:13.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:33:13.003 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T14:33:13.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:13.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T14:33:13.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:14.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:14.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:14.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:33:14.327 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T14:33:14.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:14.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T14:33:14.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:15.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:15.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:15.738 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T14:33:15.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:33:15.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:15.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T14:33:16.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:17.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:17.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:17.191 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T14:33:17.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:33:17.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:17.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T14:33:17.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:18.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:18.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:18.620 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T14:33:18.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:33:18.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:18.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T14:33:19.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:20.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:20.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:20.048 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T14:33:20.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:33:20.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:20.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T14:33:20.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:21.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:21.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:21.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:33:21.602 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T14:33:21.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:21.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T14:33:22.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:23.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:23.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:23.380 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T14:33:23.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:33:23.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:23.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T14:33:24.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:25.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:25.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:25.009 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T14:33:25.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:33:25.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:25.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T14:33:25.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:26.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:26.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:26.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T14:33:26.329 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T14:33:26.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:26.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T14:33:26.688 INFO:tasks.workunit.client.0.smithi046.stdout:osd.6 up in weight 1 up_from 34 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/53403,v1:127.0.0.1:6851/53403] [v2:127.0.0.1:6852/53403,v1:127.0.0.1:6853/53403] exists,up 54942822-d1c1-498f-a68e-8984f0b32f6a 2024-09-18T14:33:26.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:33:26.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:33:26.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:33:26.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:33:26.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 7 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2024-09-18T14:33:26.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-09-18T14:33:26.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:33:26.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-09-18T14:33:26.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:33:26.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/7 2024-09-18T14:33:26.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 ' 2024-09-18T14:33:26.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:33:26.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:33:26.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:33:26.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/7' 2024-09-18T14:33:26.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/7/journal' 2024-09-18T14:33:26.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:33:26.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:33:26.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-18T14:33:26.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:33:26.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:33:26.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:33:26.692 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:33:26.692 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:33:26.692 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:33:26.694 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:33:26.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' 2024-09-18T14:33:26.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:33:26.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:33:26.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:33:26.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-18T14:33:26.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-18T14:33:26.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:33:26.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:33:26.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:33:26.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:33:26.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:33:26.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10' 2024-09-18T14:33:26.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/7 2024-09-18T14:33:26.703 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:33:26.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=817430db-2a68-4f06-be9a-b4f76517226e 2024-09-18T14:33:26.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 817430db-2a68-4f06-be9a-b4f76517226e' 2024-09-18T14:33:26.705 INFO:tasks.workunit.client.0.smithi046.stdout:add osd7 817430db-2a68-4f06-be9a-b4f76517226e 2024-09-18T14:33:26.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:33:26.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC25Opmrb3BKhAArabPdQNAvOf9WS1pd8t1iQ== 2024-09-18T14:33:26.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC25Opmrb3BKhAArabPdQNAvOf9WS1pd8t1iQ=="}' 2024-09-18T14:33:26.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 817430db-2a68-4f06-be9a-b4f76517226e -i td/osd-recovery-scrub/7/new.json 2024-09-18T14:33:27.219 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:33:27.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/7/new.json 2024-09-18T14:33:27.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --mkfs --key AQC25Opmrb3BKhAArabPdQNAvOf9WS1pd8t1iQ== --osd-uuid 817430db-2a68-4f06-be9a-b4f76517226e 2024-09-18T14:33:27.273 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:33:27.265+0000 7fa537f6b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:33:27.389 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:33:27.381+0000 7fa537f6b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:33:27.391 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:33:27.381+0000 7fa537f6b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:33:27.392 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:33:27.385+0000 7fa537f6b540 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-09-18T14:33:27.392 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:33:27.385+0000 7fa537f6b540 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-09-18T14:33:27.393 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:33:27.385+0000 7fa537f6b540 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-09-18T14:33:27.425 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:33:27.417+0000 7fa537f6b540 -1 bluestore(td/osd-recovery-scrub/7) _read_fsid unparsable uuid 2024-09-18T14:33:33.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/7/keyring 2024-09-18T14:33:33.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:33:33.856 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd7 key to auth repository 2024-09-18T14:33:33.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-09-18T14:33:33.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:33:34.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-09-18T14:33:34.241 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.7 2024-09-18T14:33:34.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:33:34.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=e945a5cd-13c1-4bc7-a06c-85809a6418a7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 2024-09-18T14:33:34.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:33:34.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:33:34.273 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:33:34.265+0000 7ff10768f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:33:34.275 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:33:34.265+0000 7ff10768f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:33:34.277 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:33:34.269+0000 7ff10768f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:33:34.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-09-18T14:33:34.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:33:34.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-09-18T14:33:34.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:33:34.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:33:34.594 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:33:34.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:34.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:33:34.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:34.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T14:33:34.835 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:33:34.825+0000 7ff10768f540 -1 Falling back to public interface 2024-09-18T14:33:34.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:35.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:35.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:35.906 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:33:35.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:33:35.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:35.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T14:33:36.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:37.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:37.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:37.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:33:37.242 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:33:37.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:37.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T14:33:37.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:37.630 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:33:37.609+0000 7ff10768f540 -1 osd.7 0 log_to_monitors true 2024-09-18T14:33:38.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:38.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:38.545 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:33:38.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:33:38.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:38.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T14:33:38.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:39.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:39.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:39.933 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:33:39.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:33:39.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:39.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T14:33:40.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:41.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:41.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:41.330 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:33:41.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:33:41.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:41.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T14:33:41.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:42.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:42.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:42.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:33:42.654 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:33:42.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:42.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T14:33:43.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:44.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:44.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:44.070 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:33:44.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:33:44.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:44.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T14:33:44.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:45.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:45.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:45.487 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T14:33:45.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:33:45.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:45.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T14:33:45.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:46.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:46.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:46.801 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T14:33:46.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:33:46.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:46.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T14:33:47.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:48.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:48.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:48.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:33:48.117 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T14:33:48.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:48.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T14:33:48.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:49.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:49.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:49.526 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T14:33:49.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:33:49.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:49.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T14:33:49.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:50.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:50.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:50.823 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T14:33:50.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:33:50.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:50.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T14:33:51.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:52.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:52.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:52.452 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T14:33:52.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:33:52.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:52.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T14:33:52.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:53.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:53.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:53.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:33:53.948 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T14:33:53.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:53.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T14:33:54.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:55.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:55.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:55.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:33:55.593 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T14:33:55.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:55.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T14:33:56.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:57.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:57.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:57.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:33:57.440 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T14:33:57.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:57.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T14:33:58.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:59.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:59.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:59.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:33:59.125 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T14:33:59.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:59.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T14:33:59.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:34:00.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:34:00.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:34:00.951 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T14:34:00.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:34:00.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:34:00.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T14:34:01.342 INFO:tasks.workunit.client.0.smithi046.stdout:osd.7 up in weight 1 up_from 39 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/55208,v1:127.0.0.1:6859/55208] [v2:127.0.0.1:6860/55208,v1:127.0.0.1:6861/55208] exists,up 817430db-2a68-4f06-be9a-b4f76517226e 2024-09-18T14:34:01.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:34:01.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:34:01.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:34:01.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:248: TEST_recovery_scrub_2: create_pool test 32 32 2024-09-18T14:34:01.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 32 32 2024-09-18T14:34:01.973 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'test' created 2024-09-18T14:34:01.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:34:02.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: wait_for_clean 2024-09-18T14:34:02.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:34:02.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:34:02.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:34:02.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T14:34:02.988 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:34:02.989 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:34:02.989 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:34:02.989 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:34:02.989 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:34:02.990 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:34:03.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:34:03.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:34:03.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:34:03.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:34:03.196 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:34:03.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:34:03.524 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:34:03.524 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T14:34:03.524 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T14:34:03.524 INFO:tasks.workunit.client.0.smithi046.stderr:4 2024-09-18T14:34:03.524 INFO:tasks.workunit.client.0.smithi046.stderr:5 2024-09-18T14:34:03.525 INFO:tasks.workunit.client.0.smithi046.stderr:6 2024-09-18T14:34:03.525 INFO:tasks.workunit.client.0.smithi046.stderr:7' 2024-09-18T14:34:03.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:34:03.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:34:03.525 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:34:03.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836534 2024-09-18T14:34:03.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836534 2024-09-18T14:34:03.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836534' 2024-09-18T14:34:03.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:34:03.847 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:34:04.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673008 2024-09-18T14:34:04.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673008 2024-09-18T14:34:04.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836534 1-42949673008' 2024-09-18T14:34:04.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:34:04.110 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:34:04.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509478 2024-09-18T14:34:04.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509478 2024-09-18T14:34:04.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836534 1-42949673008 2-64424509478' 2024-09-18T14:34:04.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:34:04.441 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T14:34:04.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345957 2024-09-18T14:34:04.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345957 2024-09-18T14:34:04.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836534 1-42949673008 2-64424509478 3-85899345957' 2024-09-18T14:34:04.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:34:04.639 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T14:34:04.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=103079215133 2024-09-18T14:34:04.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 103079215133 2024-09-18T14:34:04.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836534 1-42949673008 2-64424509478 3-85899345957 4-103079215133' 2024-09-18T14:34:04.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:34:04.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T14:34:05.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051605 2024-09-18T14:34:05.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051605 2024-09-18T14:34:05.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836534 1-42949673008 2-64424509478 3-85899345957 4-103079215133 5-124554051605' 2024-09-18T14:34:05.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:34:05.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T14:34:05.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=146028888076 2024-09-18T14:34:05.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 146028888076 2024-09-18T14:34:05.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836534 1-42949673008 2-64424509478 3-85899345957 4-103079215133 5-124554051605 6-146028888076' 2024-09-18T14:34:05.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:34:05.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T14:34:05.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=167503724553 2024-09-18T14:34:05.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 167503724553 2024-09-18T14:34:05.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836534 1-42949673008 2-64424509478 3-85899345957 4-103079215133 5-124554051605 6-146028888076 7-167503724553' 2024-09-18T14:34:05.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:34:05.377 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:34:05.377 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836534 2024-09-18T14:34:05.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:34:05.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836534 2024-09-18T14:34:05.380 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:34:05.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836534 2024-09-18T14:34:05.382 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836534 2024-09-18T14:34:05.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2024-09-18T14:34:05.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:34:05.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836534 -lt 21474836534 2024-09-18T14:34:05.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:34:05.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673008 2024-09-18T14:34:05.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:34:05.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:34:05.704 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673008 2024-09-18T14:34:05.704 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:34:05.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673008 2024-09-18T14:34:05.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673008' 2024-09-18T14:34:05.706 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949673008 2024-09-18T14:34:05.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:34:06.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673007 -lt 42949673008 2024-09-18T14:34:06.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:34:07.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:34:07.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:34:07.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673008 -lt 42949673008 2024-09-18T14:34:07.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:34:07.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509478 2024-09-18T14:34:07.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:34:07.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:34:07.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509478 2024-09-18T14:34:07.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:34:07.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509478 2024-09-18T14:34:07.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509478' 2024-09-18T14:34:07.409 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509478 2024-09-18T14:34:07.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:34:07.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509478 -lt 64424509478 2024-09-18T14:34:07.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:34:07.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345957 2024-09-18T14:34:07.762 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:34:07.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T14:34:07.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345957 2024-09-18T14:34:07.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:34:07.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345957 2024-09-18T14:34:07.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345957' 2024-09-18T14:34:07.767 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.3 seq 85899345957 2024-09-18T14:34:07.767 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T14:34:08.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345957 -lt 85899345957 2024-09-18T14:34:08.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:34:08.101 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-103079215133 2024-09-18T14:34:08.102 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:34:08.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T14:34:08.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-103079215133 2024-09-18T14:34:08.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:34:08.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=103079215133 2024-09-18T14:34:08.107 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.4 seq 103079215133 2024-09-18T14:34:08.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 103079215133' 2024-09-18T14:34:08.107 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T14:34:08.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 103079215134 -lt 103079215133 2024-09-18T14:34:08.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:34:08.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-124554051605 2024-09-18T14:34:08.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:34:08.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T14:34:08.498 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-124554051605 2024-09-18T14:34:08.498 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:34:08.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051605 2024-09-18T14:34:08.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 124554051605' 2024-09-18T14:34:08.501 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.5 seq 124554051605 2024-09-18T14:34:08.501 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T14:34:08.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051606 -lt 124554051605 2024-09-18T14:34:08.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:34:08.827 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-146028888076 2024-09-18T14:34:08.827 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:34:08.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T14:34:08.830 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-146028888076 2024-09-18T14:34:08.830 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:34:08.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=146028888076 2024-09-18T14:34:08.832 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.6 seq 146028888076 2024-09-18T14:34:08.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 146028888076' 2024-09-18T14:34:08.832 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T14:34:09.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 146028888077 -lt 146028888076 2024-09-18T14:34:09.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:34:09.211 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-167503724553 2024-09-18T14:34:09.211 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:34:09.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T14:34:09.215 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-167503724553 2024-09-18T14:34:09.215 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:34:09.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=167503724553 2024-09-18T14:34:09.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 167503724553' 2024-09-18T14:34:09.217 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.7 seq 167503724553 2024-09-18T14:34:09.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T14:34:09.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 167503724554 -lt 167503724553 2024-09-18T14:34:09.553 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:34:09.553 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:34:09.554 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:34:09.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-09-18T14:34:09.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:34:09.902 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:34:09.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:34:09.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:34:09.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:34:09.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:34:09.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:34:10.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-09-18T14:34:10.260 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:34:10.260 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:34:10.260 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:34:10.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-09-18T14:34:10.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:34:10.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:34:10.620 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:250: TEST_recovery_scrub_2: ceph osd dump 2024-09-18T14:34:10.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:250: TEST_recovery_scrub_2: grep '^pool.*['\'']test['\'']' 2024-09-18T14:34:10.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:250: TEST_recovery_scrub_2: awk '{ print $2 }' 2024-09-18T14:34:10.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:250: TEST_recovery_scrub_2: poolid=1 2024-09-18T14:34:10.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:252: TEST_recovery_scrub_2: dd if=/dev/urandom of=testdata.31256 bs=1M count=50 2024-09-18T14:34:11.125 INFO:tasks.workunit.client.0.smithi046.stderr:50+0 records in 2024-09-18T14:34:11.126 INFO:tasks.workunit.client.0.smithi046.stderr:50+0 records out 2024-09-18T14:34:11.126 INFO:tasks.workunit.client.0.smithi046.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.1546 s, 339 MB/s 2024-09-18T14:34:11.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: seq 1 40 2024-09-18T14:34:11.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:11.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj1 testdata.31256 2024-09-18T14:34:12.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:12.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj2 testdata.31256 2024-09-18T14:34:14.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:14.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj3 testdata.31256 2024-09-18T14:34:15.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:15.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj4 testdata.31256 2024-09-18T14:34:17.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:17.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj5 testdata.31256 2024-09-18T14:34:18.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:18.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj6 testdata.31256 2024-09-18T14:34:20.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:20.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj7 testdata.31256 2024-09-18T14:34:21.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:21.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj8 testdata.31256 2024-09-18T14:34:23.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:23.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj9 testdata.31256 2024-09-18T14:34:24.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:24.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj10 testdata.31256 2024-09-18T14:34:25.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:25.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj11 testdata.31256 2024-09-18T14:34:27.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:27.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj12 testdata.31256 2024-09-18T14:34:28.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:28.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj13 testdata.31256 2024-09-18T14:34:30.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:30.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj14 testdata.31256 2024-09-18T14:34:31.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:31.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj15 testdata.31256 2024-09-18T14:34:33.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:33.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj16 testdata.31256 2024-09-18T14:34:35.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:35.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj17 testdata.31256 2024-09-18T14:34:36.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:36.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj18 testdata.31256 2024-09-18T14:34:37.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:37.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj19 testdata.31256 2024-09-18T14:34:39.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:39.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj20 testdata.31256 2024-09-18T14:34:40.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:40.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj21 testdata.31256 2024-09-18T14:34:42.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:42.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj22 testdata.31256 2024-09-18T14:34:43.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:43.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj23 testdata.31256 2024-09-18T14:34:45.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:45.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj24 testdata.31256 2024-09-18T14:34:47.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:47.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj25 testdata.31256 2024-09-18T14:34:48.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:48.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj26 testdata.31256 2024-09-18T14:34:49.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:49.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj27 testdata.31256 2024-09-18T14:34:51.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:51.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj28 testdata.31256 2024-09-18T14:34:52.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:52.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj29 testdata.31256 2024-09-18T14:34:54.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:54.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj30 testdata.31256 2024-09-18T14:34:55.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:55.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj31 testdata.31256 2024-09-18T14:34:57.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:57.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj32 testdata.31256 2024-09-18T14:34:58.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:34:58.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj33 testdata.31256 2024-09-18T14:35:00.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:35:00.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj34 testdata.31256 2024-09-18T14:35:01.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:35:01.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj35 testdata.31256 2024-09-18T14:35:03.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:35:03.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj36 testdata.31256 2024-09-18T14:35:04.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:35:04.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj37 testdata.31256 2024-09-18T14:35:05.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:35:05.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj38 testdata.31256 2024-09-18T14:35:07.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:35:07.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj39 testdata.31256 2024-09-18T14:35:08.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:253: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-18T14:35:08.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:255: TEST_recovery_scrub_2: rados -p test put obj40 testdata.31256 2024-09-18T14:35:10.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:257: TEST_recovery_scrub_2: rm -f testdata.31256 2024-09-18T14:35:10.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:259: TEST_recovery_scrub_2: ceph osd pool set test size 3 2024-09-18T14:35:10.708 INFO:tasks.workunit.client.0.smithi046.stderr:set pool 1 size to 3 2024-09-18T14:35:10.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: ceph pg dump pgs 2024-09-18T14:35:11.236 INFO:tasks.workunit.client.0.smithi046.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:35:11.237 INFO:tasks.workunit.client.0.smithi046.stdout:1.1f 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-09-18T14:34:04.427493+0000 46'26 46:37 [5] 5 [5] 5 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:22:09.642044+0000 0 0 2024-09-18T14:35:11.237 INFO:tasks.workunit.client.0.smithi046.stdout:1.1e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-18T14:34:04.427078+0000 46'13 46:24 [3] 3 [3] 3 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:38:13.526151+0000 0 0 2024-09-18T14:35:11.237 INFO:tasks.workunit.client.0.smithi046.stdout:1.1d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-18T14:34:03.912174+0000 46'13 46:25 [1] 1 [1] 1 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:45:38.858737+0000 0 0 2024-09-18T14:35:11.237 INFO:tasks.workunit.client.0.smithi046.stdout:1.b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-18T14:34:03.769294+0000 46'13 46:25 [1] 1 [1] 1 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T22:21:56.759681+0000 0 0 2024-09-18T14:35:11.237 INFO:tasks.workunit.client.0.smithi046.stdout:1.a 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-18T14:34:03.769536+0000 46'13 46:25 [1] 1 [1] 1 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:37:55.657054+0000 0 0 2024-09-18T14:35:11.237 INFO:tasks.workunit.client.0.smithi046.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:34:03.911447+0000 0'0 45:12 [1] 1 [1] 1 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:13:56.946683+0000 0 0 2024-09-18T14:35:11.237 INFO:tasks.workunit.client.0.smithi046.stdout:1.8 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-09-18T14:34:04.425791+0000 46'26 46:37 [5] 5 [5] 5 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:26:19.704917+0000 0 0 2024-09-18T14:35:11.237 INFO:tasks.workunit.client.0.smithi046.stdout:1.7 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-18T14:34:03.091940+0000 46'13 46:23 [6] 6 [6] 6 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:37:13.386829+0000 0 0 2024-09-18T14:35:11.237 INFO:tasks.workunit.client.0.smithi046.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:34:03.092151+0000 0'0 45:10 [6] 6 [6] 6 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:18:22.585053+0000 0 0 2024-09-18T14:35:11.238 INFO:tasks.workunit.client.0.smithi046.stdout:1.5 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-09-18T14:34:03.299249+0000 46'39 46:49 [4] 4 [4] 4 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:49:04.091625+0000 0 0 2024-09-18T14:35:11.238 INFO:tasks.workunit.client.0.smithi046.stdout:1.4 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-09-18T14:34:04.425502+0000 46'52 46:63 [3] 3 [3] 3 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:26:27.266398+0000 0 0 2024-09-18T14:35:11.238 INFO:tasks.workunit.client.0.smithi046.stdout:1.3 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-09-18T14:34:03.040371+0000 46'52 46:62 [4] 4 [4] 4 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:05:37.700502+0000 0 0 2024-09-18T14:35:11.238 INFO:tasks.workunit.client.0.smithi046.stdout:1.2 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-18T14:34:04.425254+0000 46'13 46:24 [0] 0 [0] 0 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:44:55.209633+0000 0 0 2024-09-18T14:35:11.238 INFO:tasks.workunit.client.0.smithi046.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:34:04.323060+0000 0'0 45:11 [3] 3 [3] 3 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:33:33.911345+0000 0 0 2024-09-18T14:35:11.238 INFO:tasks.workunit.client.0.smithi046.stdout:1.1 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-18T14:34:04.426721+0000 46'13 46:24 [5] 5 [5] 5 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-20T02:15:11.285068+0000 0 0 2024-09-18T14:35:11.238 INFO:tasks.workunit.client.0.smithi046.stdout:1.1c 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-18T14:34:04.427067+0000 46'13 46:24 [0] 0 [0] 0 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:19:46.987464+0000 0 0 2024-09-18T14:35:11.238 INFO:tasks.workunit.client.0.smithi046.stdout:1.0 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-18T14:34:03.040058+0000 46'13 46:24 [7] 7 [7] 7 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-20T02:09:22.141900+0000 0 0 2024-09-18T14:35:11.238 INFO:tasks.workunit.client.0.smithi046.stdout:1.d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-18T14:34:03.298678+0000 46'13 46:24 [7] 7 [7] 7 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:28:05.512687+0000 0 0 2024-09-18T14:35:11.238 INFO:tasks.workunit.client.0.smithi046.stdout:1.e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-18T14:34:03.912034+0000 46'13 46:25 [1] 1 [1] 1 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:50:56.155327+0000 0 0 2024-09-18T14:35:11.239 INFO:tasks.workunit.client.0.smithi046.stdout:1.f 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-18T14:34:03.092374+0000 46'13 46:23 [6] 6 [6] 6 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:08:41.446568+0000 0 0 2024-09-18T14:35:11.239 INFO:tasks.workunit.client.0.smithi046.stdout:1.10 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-18T14:34:04.427150+0000 46'13 46:24 [5] 5 [5] 5 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:21:32.808557+0000 0 0 2024-09-18T14:35:11.239 INFO:tasks.workunit.client.0.smithi046.stdout:1.11 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-18T14:34:04.425855+0000 46'13 46:24 [0] 0 [0] 0 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-20T02:14:56.049745+0000 0 0 2024-09-18T14:35:11.239 INFO:tasks.workunit.client.0.smithi046.stdout:1.12 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-09-18T14:34:04.276377+0000 46'26 46:37 [0] 0 [0] 0 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:57:48.080304+0000 0 0 2024-09-18T14:35:11.239 INFO:tasks.workunit.client.0.smithi046.stdout:1.13 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-18T14:34:03.298771+0000 46'13 46:23 [4] 4 [4] 4 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:20:54.923884+0000 0 0 2024-09-18T14:35:11.239 INFO:tasks.workunit.client.0.smithi046.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:34:04.276389+0000 0'0 45:11 [2] 2 [2] 2 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:56:51.886758+0000 0 0 2024-09-18T14:35:11.239 INFO:tasks.workunit.client.0.smithi046.stdout:1.15 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-09-18T14:34:03.911821+0000 46'39 46:51 [1] 1 [1] 1 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:06:10.309706+0000 0 0 2024-09-18T14:35:11.239 INFO:tasks.workunit.client.0.smithi046.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:34:03.298629+0000 0'0 45:10 [6] 6 [6] 6 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:41:04.124039+0000 0 0 2024-09-18T14:35:11.239 INFO:tasks.workunit.client.0.smithi046.stdout:1.17 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-18T14:34:04.426897+0000 46'13 46:24 [0] 0 [0] 0 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:44:53.504340+0000 0 0 2024-09-18T14:35:11.239 INFO:tasks.workunit.client.0.smithi046.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:34:04.276366+0000 0'0 45:11 [5] 5 [5] 5 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:10:47.747826+0000 0 0 2024-09-18T14:35:11.239 INFO:tasks.workunit.client.0.smithi046.stdout:1.19 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-18T14:34:04.426771+0000 46'13 46:24 [3] 3 [3] 3 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:20:27.833933+0000 0 0 2024-09-18T14:35:11.239 INFO:tasks.workunit.client.0.smithi046.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:34:03.912041+0000 0'0 45:12 [1] 1 [1] 1 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:20:23.697578+0000 0 0 2024-09-18T14:35:11.240 INFO:tasks.workunit.client.0.smithi046.stdout:1.1b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-18T14:34:04.425981+0000 46'13 46:24 [0] 0 [0] 0 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:49:14.351467+0000 0 0 2024-09-18T14:35:11.240 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-09-18T14:35:11.240 INFO:tasks.workunit.client.0.smithi046.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:35:11.240 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:35:11.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:264: TEST_recovery_scrub_2: count=0 2024-09-18T14:35:11.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:265: TEST_recovery_scrub_2: true 2024-09-18T14:35:11.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-09-18T14:35:11.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:270: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-09-18T14:35:11.252 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:270: TEST_recovery_scrub_2: grep recovering 2024-09-18T14:35:11.252 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:270: TEST_recovery_scrub_2: wc -l 2024-09-18T14:35:11.787 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:35:11.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: test 0 -ge 2 2024-09-18T14:35:11.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:273: TEST_recovery_scrub_2: sleep 2 2024-09-18T14:35:13.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:274: TEST_recovery_scrub_2: test 0 -eq 10 2024-09-18T14:35:13.801 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:279: TEST_recovery_scrub_2: expr 0 + 1 2024-09-18T14:35:13.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:279: TEST_recovery_scrub_2: count=1 2024-09-18T14:35:13.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:265: TEST_recovery_scrub_2: true 2024-09-18T14:35:13.806 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-09-18T14:35:13.806 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:270: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-09-18T14:35:13.806 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:270: TEST_recovery_scrub_2: grep recovering 2024-09-18T14:35:13.806 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:270: TEST_recovery_scrub_2: wc -l 2024-09-18T14:35:14.107 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:35:14.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: test 0 -ge 2 2024-09-18T14:35:14.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:273: TEST_recovery_scrub_2: sleep 2 2024-09-18T14:35:16.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:274: TEST_recovery_scrub_2: test 1 -eq 10 2024-09-18T14:35:16.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:279: TEST_recovery_scrub_2: expr 1 + 1 2024-09-18T14:35:16.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:279: TEST_recovery_scrub_2: count=2 2024-09-18T14:35:16.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:265: TEST_recovery_scrub_2: true 2024-09-18T14:35:16.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-09-18T14:35:16.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:270: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-09-18T14:35:16.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:270: TEST_recovery_scrub_2: grep recovering 2024-09-18T14:35:16.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:270: TEST_recovery_scrub_2: wc -l 2024-09-18T14:35:16.603 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:35:16.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: test 0 -ge 2 2024-09-18T14:35:16.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:273: TEST_recovery_scrub_2: sleep 2 2024-09-18T14:35:18.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:274: TEST_recovery_scrub_2: test 2 -eq 10 2024-09-18T14:35:18.617 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:279: TEST_recovery_scrub_2: expr 2 + 1 2024-09-18T14:35:18.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:279: TEST_recovery_scrub_2: count=3 2024-09-18T14:35:18.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:265: TEST_recovery_scrub_2: true 2024-09-18T14:35:18.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-09-18T14:35:18.622 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:270: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-09-18T14:35:18.622 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:270: TEST_recovery_scrub_2: grep recovering 2024-09-18T14:35:18.622 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:270: TEST_recovery_scrub_2: wc -l 2024-09-18T14:35:18.923 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:35:18.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: test 2 -ge 2 2024-09-18T14:35:18.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:271: TEST_recovery_scrub_2: break 2024-09-18T14:35:18.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:281: TEST_recovery_scrub_2: ceph pg dump pgs 2024-09-18T14:35:19.224 INFO:tasks.workunit.client.0.smithi046.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:35:19.225 INFO:tasks.workunit.client.0.smithi046.stdout:1.1f 2 0 4 0 0 104857600 0 0 26 0 26 active+recovering+degraded 2024-09-18T14:35:12.619587+0000 46'26 48:49 [5,0,2] 5 [5,0,2] 5 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:35:55.497168+0000 0 0 2024-09-18T14:35:19.225 INFO:tasks.workunit.client.0.smithi046.stdout:1.1e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-18T14:35:14.571857+0000 46'13 49:35 [3,0,7] 3 [3,0,7] 3 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T22:07:57.396742+0000 0 0 2024-09-18T14:35:19.225 INFO:tasks.workunit.client.0.smithi046.stdout:1.1d 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-09-18T14:35:16.236391+0000 46'26 49:49 [1,3,0] 1 [1,3,0] 1 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:43:37.178662+0000 0 0 2024-09-18T14:35:19.225 INFO:tasks.workunit.client.0.smithi046.stdout:1.b 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-18T14:35:15.451864+0000 46'13 49:36 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:14:53.049267+0000 0 0 2024-09-18T14:35:19.225 INFO:tasks.workunit.client.0.smithi046.stdout:1.a 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-18T14:35:15.451708+0000 46'13 49:36 [1,6,2] 1 [1,6,2] 1 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:41:48.907960+0000 0 0 2024-09-18T14:35:19.225 INFO:tasks.workunit.client.0.smithi046.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:35:15.009452+0000 0'0 49:25 [1,4,5] 1 [1,4,5] 1 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:48:57.037479+0000 0 0 2024-09-18T14:35:19.225 INFO:tasks.workunit.client.0.smithi046.stdout:1.8 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-09-18T14:35:13.134869+0000 46'26 48:48 [5,4,3] 5 [5,4,3] 5 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:06:37.625586+0000 0 0 2024-09-18T14:35:19.225 INFO:tasks.workunit.client.0.smithi046.stdout:1.7 1 0 2 0 0 52428800 0 0 13 0 13 active+recovering+degraded 2024-09-18T14:35:12.376293+0000 46'13 48:35 [6,7,3] 6 [6,7,3] 6 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:27:23.558408+0000 0 0 2024-09-18T14:35:19.225 INFO:tasks.workunit.client.0.smithi046.stdout:1.6 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-18T14:35:12.374475+0000 46'13 48:34 [6,7,3] 6 [6,7,3] 6 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:25:40.205564+0000 0 0 2024-09-18T14:35:19.225 INFO:tasks.workunit.client.0.smithi046.stdout:1.5 3 0 6 0 0 157286400 0 0 39 0 39 active+recovery_wait+degraded 2024-09-18T14:35:12.618521+0000 46'39 48:60 [4,2,6] 4 [4,2,6] 4 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:29:01.783995+0000 0 0 2024-09-18T14:35:19.225 INFO:tasks.workunit.client.0.smithi046.stdout:1.4 4 0 8 0 0 209715200 0 0 52 0 52 active+recovery_wait+degraded 2024-09-18T14:35:13.079559+0000 46'52 48:74 [3,0,6] 3 [3,0,6] 3 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:26:34.057302+0000 0 0 2024-09-18T14:35:19.226 INFO:tasks.workunit.client.0.smithi046.stdout:1.3 4 0 8 0 0 209715200 0 0 52 0 52 active+recovery_wait+degraded 2024-09-18T14:35:13.995535+0000 46'52 49:73 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-20T02:03:55.301488+0000 0 0 2024-09-18T14:35:19.226 INFO:tasks.workunit.client.0.smithi046.stdout:1.2 1 0 2 0 0 52428800 0 0 13 0 13 activating+degraded 2024-09-18T14:35:13.007418+0000 46'13 48:30 [0,7,1] 0 [0,7,1] 0 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:29:50.307079+0000 0 0 2024-09-18T14:35:19.226 INFO:tasks.workunit.client.0.smithi046.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:35:12.268385+0000 0'0 48:24 [3,0,4] 3 [3,0,4] 3 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:45:50.878702+0000 0 0 2024-09-18T14:35:19.226 INFO:tasks.workunit.client.0.smithi046.stdout:1.1 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-18T14:35:13.135028+0000 46'13 48:35 [5,0,4] 5 [5,0,4] 5 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:27:14.105172+0000 0 0 2024-09-18T14:35:19.226 INFO:tasks.workunit.client.0.smithi046.stdout:1.1c 1 0 2 0 0 52428800 0 0 13 0 13 activating+degraded 2024-09-18T14:35:12.616973+0000 46'13 48:31 [0,3,4] 0 [0,3,4] 0 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:17:19.352552+0000 0 0 2024-09-18T14:35:19.226 INFO:tasks.workunit.client.0.smithi046.stdout:1.0 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-18T14:35:13.995206+0000 46'13 49:35 [7,0,2] 7 [7,0,2] 7 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:25:11.177453+0000 0 0 2024-09-18T14:35:19.226 INFO:tasks.workunit.client.0.smithi046.stdout:1.d 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-18T14:35:13.078047+0000 46'13 48:35 [7,4,2] 7 [7,4,2] 7 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:03:30.881271+0000 0 0 2024-09-18T14:35:19.226 INFO:tasks.workunit.client.0.smithi046.stdout:1.e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-18T14:35:15.451705+0000 46'13 49:36 [1,4,2] 1 [1,4,2] 1 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:24:24.519366+0000 0 0 2024-09-18T14:35:19.227 INFO:tasks.workunit.client.0.smithi046.stdout:1.f 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-18T14:35:13.991148+0000 46'13 49:34 [6,0,3] 6 [6,0,3] 6 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:18:32.538945+0000 0 0 2024-09-18T14:35:19.227 INFO:tasks.workunit.client.0.smithi046.stdout:1.10 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-18T14:35:13.009499+0000 46'13 48:35 [5,7,6] 5 [5,7,6] 5 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:17:23.248583+0000 0 0 2024-09-18T14:35:19.227 INFO:tasks.workunit.client.0.smithi046.stdout:1.11 1 0 2 0 0 52428800 0 0 13 0 13 activating+degraded 2024-09-18T14:35:12.615713+0000 46'13 48:30 [0,1,3] 0 [0,1,3] 0 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:18:48.400344+0000 0 0 2024-09-18T14:35:19.227 INFO:tasks.workunit.client.0.smithi046.stdout:1.12 2 0 4 0 0 104857600 0 0 26 0 26 activating+degraded 2024-09-18T14:35:12.496027+0000 46'26 48:43 [0,1,6] 0 [0,1,6] 0 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:57:40.291932+0000 0 0 2024-09-18T14:35:19.227 INFO:tasks.workunit.client.0.smithi046.stdout:1.13 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-18T14:35:12.617154+0000 46'13 48:34 [4,6,3] 4 [4,6,3] 4 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:47:09.280644+0000 0 0 2024-09-18T14:35:19.227 INFO:tasks.workunit.client.0.smithi046.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:35:14.305031+0000 0'0 49:24 [2,0,4] 2 [2,0,4] 2 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:22:13.616134+0000 0 0 2024-09-18T14:35:19.227 INFO:tasks.workunit.client.0.smithi046.stdout:1.15 3 0 6 0 0 157286400 0 0 39 0 39 active+recovery_wait+degraded 2024-09-18T14:35:16.236517+0000 46'39 49:62 [1,5,3] 1 [1,5,3] 1 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:19:30.443010+0000 0 0 2024-09-18T14:35:19.227 INFO:tasks.workunit.client.0.smithi046.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:35:12.115381+0000 0'0 48:23 [6,0,7] 6 [6,0,7] 6 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:48:38.151598+0000 0 0 2024-09-18T14:35:19.227 INFO:tasks.workunit.client.0.smithi046.stdout:1.17 1 0 2 0 0 52428800 0 0 13 0 13 activating+degraded 2024-09-18T14:35:12.615208+0000 46'13 48:31 [0,4,5] 0 [0,4,5] 0 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:19:46.346897+0000 0 0 2024-09-18T14:35:19.227 INFO:tasks.workunit.client.0.smithi046.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:35:12.616915+0000 0'0 48:24 [5,2,3] 5 [5,2,3] 5 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:08:32.454733+0000 0 0 2024-09-18T14:35:19.227 INFO:tasks.workunit.client.0.smithi046.stdout:1.19 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-18T14:35:14.571534+0000 46'13 49:35 [3,0,4] 3 [3,0,4] 3 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:42:55.974422+0000 0 0 2024-09-18T14:35:19.227 INFO:tasks.workunit.client.0.smithi046.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:35:15.008623+0000 0'0 49:25 [1,7,4] 1 [1,7,4] 1 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:00:47.797961+0000 0 0 2024-09-18T14:35:19.227 INFO:tasks.workunit.client.0.smithi046.stdout:1.1b 1 0 2 0 0 52428800 0 0 13 0 13 activating+degraded 2024-09-18T14:35:12.266912+0000 46'13 48:31 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-18T14:34:01.722932+0000 0'0 2024-09-18T14:34:01.722932+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:13:07.669677+0000 0 0 2024-09-18T14:35:19.228 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-09-18T14:35:19.228 INFO:tasks.workunit.client.0.smithi046.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:35:19.228 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:35:19.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:283: TEST_recovery_scrub_2: pids= 2024-09-18T14:35:19.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:284: TEST_recovery_scrub_2: recov_scrub_count=0 2024-09-18T14:35:19.237 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: expr 32 - 1 2024-09-18T14:35:19.239 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: seq 0 31 2024-09-18T14:35:19.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 0 2024-09-18T14:35:19.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.0 2024-09-18T14:35:19.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 57898"' 2024-09-18T14:35:19.242 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 57898' 2024-09-18T14:35:19.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/57900: /' 2024-09-18T14:35:19.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.0 2024-09-18T14:35:19.243 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 1 2024-09-18T14:35:19.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1 2024-09-18T14:35:19.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 57903"' 2024-09-18T14:35:19.244 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 57903' 2024-09-18T14:35:19.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.1 2024-09-18T14:35:19.245 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 2 2024-09-18T14:35:19.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/57906: /' 2024-09-18T14:35:19.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.2 2024-09-18T14:35:19.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 57912"' 2024-09-18T14:35:19.246 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 57912' 2024-09-18T14:35:19.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.2 2024-09-18T14:35:19.248 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 3 2024-09-18T14:35:19.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/57915: /' 2024-09-18T14:35:19.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.3 2024-09-18T14:35:19.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 57917"' 2024-09-18T14:35:19.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 57917' 2024-09-18T14:35:19.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 4 2024-09-18T14:35:19.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.4 2024-09-18T14:35:19.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 57922"' 2024-09-18T14:35:19.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 57922' 2024-09-18T14:35:19.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.3 2024-09-18T14:35:19.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 5 2024-09-18T14:35:19.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.5 2024-09-18T14:35:19.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.4 2024-09-18T14:35:19.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 57928"' 2024-09-18T14:35:19.252 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 57928' 2024-09-18T14:35:19.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/57926: /' 2024-09-18T14:35:19.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 6 2024-09-18T14:35:19.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.6 2024-09-18T14:35:19.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 57933"' 2024-09-18T14:35:19.255 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 57933' 2024-09-18T14:35:19.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.256 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 7 2024-09-18T14:35:19.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.7 2024-09-18T14:35:19.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 57941"' 2024-09-18T14:35:19.258 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 57941' 2024-09-18T14:35:19.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.6 2024-09-18T14:35:19.260 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 8 2024-09-18T14:35:19.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.8 2024-09-18T14:35:19.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.7 2024-09-18T14:35:19.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 57946"' 2024-09-18T14:35:19.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 57946' 2024-09-18T14:35:19.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/57944: /' 2024-09-18T14:35:19.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.8 2024-09-18T14:35:19.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/57950: /' 2024-09-18T14:35:19.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 9 2024-09-18T14:35:19.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.9 2024-09-18T14:35:19.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 57956"' 2024-09-18T14:35:19.264 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 57956' 2024-09-18T14:35:19.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/57936: /' 2024-09-18T14:35:19.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/57923: /' 2024-09-18T14:35:19.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.5 2024-09-18T14:35:19.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/57940: /' 2024-09-18T14:35:19.266 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 10 2024-09-18T14:35:19.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.a 2024-09-18T14:35:19.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 57959"' 2024-09-18T14:35:19.267 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 57959' 2024-09-18T14:35:19.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.285 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 11 2024-09-18T14:35:19.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.b 2024-09-18T14:35:19.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 57969"' 2024-09-18T14:35:19.286 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 57969' 2024-09-18T14:35:19.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.292 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 12 2024-09-18T14:35:19.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.c 2024-09-18T14:35:19.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 57972"' 2024-09-18T14:35:19.294 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 57972' 2024-09-18T14:35:19.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.a 2024-09-18T14:35:19.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/57968: /' 2024-09-18T14:35:19.301 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 13 2024-09-18T14:35:19.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.d 2024-09-18T14:35:19.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 57976"' 2024-09-18T14:35:19.302 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 57976' 2024-09-18T14:35:19.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.9 2024-09-18T14:35:19.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/57973: /' 2024-09-18T14:35:19.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.b 2024-09-18T14:35:19.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/57980: /' 2024-09-18T14:35:19.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.c 2024-09-18T14:35:19.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/57966: /' 2024-09-18T14:35:19.321 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 14 2024-09-18T14:35:19.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.e 2024-09-18T14:35:19.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 57993"' 2024-09-18T14:35:19.322 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 57993' 2024-09-18T14:35:19.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.d 2024-09-18T14:35:19.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/57992: /' 2024-09-18T14:35:19.329 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 15 2024-09-18T14:35:19.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.f 2024-09-18T14:35:19.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 57998"' 2024-09-18T14:35:19.330 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 57998' 2024-09-18T14:35:19.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/57997: /' 2024-09-18T14:35:19.336 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 16 2024-09-18T14:35:19.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.10 2024-09-18T14:35:19.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 58000"' 2024-09-18T14:35:19.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 58000' 2024-09-18T14:35:19.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.e 2024-09-18T14:35:19.344 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 17 2024-09-18T14:35:19.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.11 2024-09-18T14:35:19.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 58011"' 2024-09-18T14:35:19.353 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 58011' 2024-09-18T14:35:19.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/58006: /' 2024-09-18T14:35:19.354 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 18 2024-09-18T14:35:19.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.12 2024-09-18T14:35:19.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 58013"' 2024-09-18T14:35:19.354 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 58013' 2024-09-18T14:35:19.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.f 2024-09-18T14:35:19.356 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 19 2024-09-18T14:35:19.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.13 2024-09-18T14:35:19.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.10 2024-09-18T14:35:19.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 58017"' 2024-09-18T14:35:19.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 58017' 2024-09-18T14:35:19.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/58008: /' 2024-09-18T14:35:19.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 20 2024-09-18T14:35:19.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.14 2024-09-18T14:35:19.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 58024"' 2024-09-18T14:35:19.359 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 58024' 2024-09-18T14:35:19.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.12 2024-09-18T14:35:19.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 21 2024-09-18T14:35:19.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.15 2024-09-18T14:35:19.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 58029"' 2024-09-18T14:35:19.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 58029' 2024-09-18T14:35:19.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.13 2024-09-18T14:35:19.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/58023: /' 2024-09-18T14:35:19.362 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 22 2024-09-18T14:35:19.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/58027: /' 2024-09-18T14:35:19.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.16 2024-09-18T14:35:19.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/58021: /' 2024-09-18T14:35:19.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 58037"' 2024-09-18T14:35:19.364 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 58037' 2024-09-18T14:35:19.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/58033: /' 2024-09-18T14:35:19.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.11 2024-09-18T14:35:19.365 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 23 2024-09-18T14:35:19.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.17 2024-09-18T14:35:19.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 58042"' 2024-09-18T14:35:19.366 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 58042' 2024-09-18T14:35:19.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.15 2024-09-18T14:35:19.370 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 24 2024-09-18T14:35:19.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.18 2024-09-18T14:35:19.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 58055"' 2024-09-18T14:35:19.370 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 58055' 2024-09-18T14:35:19.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/58047: /' 2024-09-18T14:35:19.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.17 2024-09-18T14:35:19.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/58054: /' 2024-09-18T14:35:19.371 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 25 2024-09-18T14:35:19.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.19 2024-09-18T14:35:19.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 58058"' 2024-09-18T14:35:19.376 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 58058' 2024-09-18T14:35:19.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.14 2024-09-18T14:35:19.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.16 2024-09-18T14:35:19.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/58050: /' 2024-09-18T14:35:19.381 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 26 2024-09-18T14:35:19.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1a 2024-09-18T14:35:19.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 58070"' 2024-09-18T14:35:19.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 58070' 2024-09-18T14:35:19.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/58069: /' 2024-09-18T14:35:19.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.19 2024-09-18T14:35:19.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 27 2024-09-18T14:35:19.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1b 2024-09-18T14:35:19.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 58079"' 2024-09-18T14:35:19.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 58079' 2024-09-18T14:35:19.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.1a 2024-09-18T14:35:19.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/58068: /' 2024-09-18T14:35:19.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.18 2024-09-18T14:35:19.394 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 28 2024-09-18T14:35:19.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1c 2024-09-18T14:35:19.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 58088"' 2024-09-18T14:35:19.397 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 58088' 2024-09-18T14:35:19.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/58078: /' 2024-09-18T14:35:19.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 29 2024-09-18T14:35:19.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/58093: /' 2024-09-18T14:35:19.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1d 2024-09-18T14:35:19.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 58097"' 2024-09-18T14:35:19.409 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 58097' 2024-09-18T14:35:19.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.1b 2024-09-18T14:35:19.418 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 30 2024-09-18T14:35:19.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1e 2024-09-18T14:35:19.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 58102"' 2024-09-18T14:35:19.419 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 58102' 2024-09-18T14:35:19.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:285: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-18T14:35:19.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/58095: /' 2024-09-18T14:35:19.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.1c 2024-09-18T14:35:19.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.1d 2024-09-18T14:35:19.422 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: printf %x 31 2024-09-18T14:35:19.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1f 2024-09-18T14:35:19.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T14:35:19.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T14:35:19.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 58114"' 2024-09-18T14:35:19.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 58114' 2024-09-18T14:35:19.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:289: TEST_recovery_scrub_2: wait_background_check pids 2024-09-18T14:35:19.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:206: wait_background_check: pids=' 57898 57903 57912 57917 57922 57928 57933 57941 57946 57956 57959 57969 57972 57976 57993 57998 58000 58011 58013 58017 58024 58029 58037 58042 58055 58058 58070 58079 58088 58097 58102 58114' 2024-09-18T14:35:19.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:208: wait_background_check: return_code=0 2024-09-18T14:35:19.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:19.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 57898 2024-09-18T14:35:19.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/58108: /' 2024-09-18T14:35:19.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/58113: /' 2024-09-18T14:35:19.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.1e 2024-09-18T14:35:19.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/58124: /' 2024-09-18T14:35:19.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: pg_scrub_mod 1.1f 2024-09-18T14:35:25.036 INFO:tasks.workunit.client.0.smithi046.stderr:58027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.13 2024-09-18T14:35:25.036 INFO:tasks.workunit.client.0.smithi046.stderr:58027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.13 2024-09-18T14:35:25.037 INFO:tasks.workunit.client.0.smithi046.stderr:58027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2024-09-18T14:35:25.037 INFO:tasks.workunit.client.0.smithi046.stderr:58027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:25.037 INFO:tasks.workunit.client.0.smithi046.stderr:58027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:25.037 INFO:tasks.workunit.client.0.smithi046.stderr:58027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-09-18T14:35:25.037 INFO:tasks.workunit.client.0.smithi046.stderr:58027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:25.037 INFO:tasks.workunit.client.0.smithi046.stderr:58027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.13 query 2024-09-18T14:35:25.037 INFO:tasks.workunit.client.0.smithi046.stderr:58027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:25.037 INFO:tasks.workunit.client.0.smithi046.stderr:58027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:25.037 INFO:tasks.workunit.client.0.smithi046.stderr:58027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:25.037 INFO:tasks.workunit.client.0.smithi046.stderr:58027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:25.037 INFO:tasks.workunit.client.0.smithi046.stderr:58027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.13 2024-09-18T14:35:25.037 INFO:tasks.workunit.client.0.smithi046.stderr:58027: instructing pg 1.13 on osd.4 to scrub 2024-09-18T14:35:25.038 INFO:tasks.workunit.client.0.smithi046.stderr:58027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.13") | .state' 2024-09-18T14:35:25.038 INFO:tasks.workunit.client.0.smithi046.stderr:58027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:25.038 INFO:tasks.workunit.client.0.smithi046.stderr:58027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:25.038 INFO:tasks.workunit.client.0.smithi046.stderr:58027: dumped pgs 2024-09-18T14:35:25.038 INFO:tasks.workunit.client.0.smithi046.stderr:58027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: recovery=true 2024-09-18T14:35:25.038 INFO:tasks.workunit.client.0.smithi046.stderr:58027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.13 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:25.038 INFO:tasks.workunit.client.0.smithi046.stderr:58027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.13 2024-09-18T14:35:25.038 INFO:tasks.workunit.client.0.smithi046.stderr:58027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:25.038 INFO:tasks.workunit.client.0.smithi046.stderr:58027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:25.038 INFO:tasks.workunit.client.0.smithi046.stderr:58027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:25.038 INFO:tasks.workunit.client.0.smithi046.stderr:58027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:25.038 INFO:tasks.workunit.client.0.smithi046.stderr:58027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:25.038 INFO:tasks.workunit.client.0.smithi046.stderr:58027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:25.039 INFO:tasks.workunit.client.0.smithi046.stderr:58027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.13 last_scrub_stamp 2024-09-18T14:35:25.039 INFO:tasks.workunit.client.0.smithi046.stderr:58027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2024-09-18T14:35:25.039 INFO:tasks.workunit.client.0.smithi046.stderr:58027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:25.039 INFO:tasks.workunit.client.0.smithi046.stderr:58027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-09-18T14:35:25.603 INFO:tasks.workunit.client.0.smithi046.stderr:58027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: g57940: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.6 2024-09-18T14:35:25.603 INFO:tasks.workunit.client.0.smithi046.stderr:57940: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.6 2024-09-18T14:35:25.604 INFO:tasks.workunit.client.0.smithi046.stderr:57940: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-09-18T14:35:25.604 INFO:tasks.workunit.client.0.smithi046.stderr:57940: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:25.604 INFO:tasks.workunit.client.0.smithi046.stderr:57940: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-09-18T14:35:25.604 INFO:tasks.workunit.client.0.smithi046.stderr:57940: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:25.604 INFO:tasks.workunit.client.0.smithi046.stderr:57940: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:25.604 INFO:tasks.workunit.client.0.smithi046.stderr:57940: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.6 query 2024-09-18T14:35:25.604 INFO:tasks.workunit.client.0.smithi046.stderr:57940: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:25.604 INFO:tasks.workunit.client.0.smithi046.stderr:57940: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:25.604 INFO:tasks.workunit.client.0.smithi046.stderr:57940: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:25.604 INFO:tasks.workunit.client.0.smithi046.stderr:57940: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:25.604 INFO:tasks.workunit.client.0.smithi046.stderr:57940: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.6 2024-09-18T14:35:25.605 INFO:tasks.workunit.client.0.smithi046.stderr:57940: instructing pg 1.6 on osd.6 to scrub 2024-09-18T14:35:25.605 INFO:tasks.workunit.client.0.smithi046.stderr:57940: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:25.605 INFO:tasks.workunit.client.0.smithi046.stderr:57940: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.6") | .state' 2024-09-18T14:35:25.605 INFO:tasks.workunit.client.0.smithi046.stderr:57940: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:25.605 INFO:tasks.workunit.client.0.smithi046.stderr:57940: dumped pgs 2024-09-18T14:35:25.605 INFO:tasks.workunit.client.0.smithi046.stderr:57940: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.6 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:25.605 INFO:tasks.workunit.client.0.smithi046.stderr:57940: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.6 2024-09-18T14:35:25.605 INFO:tasks.workunit.client.0.smithi046.stderr:57940: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:25.605 INFO:tasks.workunit.client.0.smithi046.stderr:57940: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:25.605 INFO:tasks.workunit.client.0.smithi046.stderr:57940: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:25.605 INFO:tasks.workunit.client.0.smithi046.stderr:57940: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:25.605 INFO:tasks.workunit.client.0.smithi046.stderr:57940: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:25.605 INFO:tasks.workunit.client.0.smithi046.stderr:57940: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:25.605 INFO:tasks.workunit.client.0.smithi046.stderr:57940: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.6 last_scrub_stamp 2024-09-18T14:35:25.606 INFO:tasks.workunit.client.0.smithi046.stderr:57940: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-09-18T14:35:25.606 INFO:tasks.workunit.client.0.smithi046.stderr:57940: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:25.606 INFO:tasks.workunit.client.0.smithi046.stderr:57940: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-09-18T14:35:25.606 INFO:tasks.workunit.client.0.smithi046.stderr:57940: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:25.606 INFO:tasks.workunit.client.0.smithi046.stderr:57940: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:25.606 INFO:tasks.workunit.client.0.smithi046.stderr:57940: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:25.606 INFO:tasks.workunit.client.0.smithi046.stderr:57940: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:25.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:26.270 INFO:tasks.workunit.client.0.smithi046.stderr:57992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.d 2024-09-18T14:35:26.270 INFO:tasks.workunit.client.0.smithi046.stderr:57992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.d 2024-09-18T14:35:26.270 INFO:tasks.workunit.client.0.smithi046.stderr:57992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2024-09-18T14:35:26.270 INFO:tasks.workunit.client.0.smithi046.stderr:57992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:26.271 INFO:tasks.workunit.client.0.smithi046.stderr:57992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-09-18T14:35:26.271 INFO:tasks.workunit.client.0.smithi046.stderr:57992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:26.271 INFO:tasks.workunit.client.0.smithi046.stderr:57992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:26.271 INFO:tasks.workunit.client.0.smithi046.stderr:57992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.d query 2024-09-18T14:35:26.271 INFO:tasks.workunit.client.0.smithi046.stderr:57992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:26.271 INFO:tasks.workunit.client.0.smithi046.stderr:57992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:26.271 INFO:tasks.workunit.client.0.smithi046.stderr:57992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:26.271 INFO:tasks.workunit.client.0.smithi046.stderr:57992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:26.271 INFO:tasks.workunit.client.0.smithi046.stderr:57992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.d 2024-09-18T14:35:26.271 INFO:tasks.workunit.client.0.smithi046.stderr:57992: instructing pg 1.d on osd.7 to scrub 2024-09-18T14:35:26.271 INFO:tasks.workunit.client.0.smithi046.stderr:57992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:26.271 INFO:tasks.workunit.client.0.smithi046.stderr:57992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.d") | .state' 2024-09-18T14:35:26.272 INFO:tasks.workunit.client.0.smithi046.stderr:57992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:26.272 INFO:tasks.workunit.client.0.smithi046.stderr:57992: dumped pgs 2024-09-18T14:35:26.272 INFO:tasks.workunit.client.0.smithi046.stderr:57992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: recovery=true 2024-09-18T14:35:26.272 INFO:tasks.workunit.client.0.smithi046.stderr:57992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.d 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:26.272 INFO:tasks.workunit.client.0.smithi046.stderr:57992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.d 2024-09-18T14:35:26.272 INFO:tasks.workunit.client.0.smithi046.stderr:57992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:26.272 INFO:tasks.workunit.client.0.smithi046.stderr:57992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:26.272 INFO:tasks.workunit.client.0.smithi046.stderr:57992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:26.272 INFO:tasks.workunit.client.0.smithi046.stderr:57992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:26.272 INFO:tasks.workunit.client.0.smithi046.stderr:57992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:26.272 INFO:tasks.workunit.client.0.smithi046.stderr:57992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:26.272 INFO:tasks.workunit.client.0.smithi046.stderr:57992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.d last_scrub_stamp 2024-09-18T14:35:26.272 INFO:tasks.workunit.client.0.smithi046.stderr:57992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2024-09-18T14:35:26.273 INFO:tasks.workunit.client.0.smithi046.stderr:57992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:26.273 INFO:tasks.workunit.client.0.smithi046.stderr:57992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-09-18T14:35:26.281 INFO:tasks.workunit.client.0.smithi046.stderr:57992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub57926: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.4 2024-09-18T14:35:26.281 INFO:tasks.workunit.client.0.smithi046.stderr:57926: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.4 2024-09-18T14:35:26.281 INFO:tasks.workunit.client.0.smithi046.stderr:57926: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-09-18T14:35:26.281 INFO:tasks.workunit.client.0.smithi046.stderr:57926: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:26.281 INFO:tasks.workunit.client.0.smithi046.stderr:57926: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:26.281 INFO:tasks.workunit.client.0.smithi046.stderr:57926: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-09-18T14:35:26.281 INFO:tasks.workunit.client.0.smithi046.stderr:57926: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:26.282 INFO:tasks.workunit.client.0.smithi046.stderr:57926: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.4 query 2024-09-18T14:35:26.282 INFO:tasks.workunit.client.0.smithi046.stderr:57926: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:26.282 INFO:tasks.workunit.client.0.smithi046.stderr:57926: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:26.282 INFO:tasks.workunit.client.0.smithi046.stderr:57926: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:26.282 INFO:tasks.workunit.client.0.smithi046.stderr:57926: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:26.282 INFO:tasks.workunit.client.0.smithi046.stderr:57926: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.4 2024-09-18T14:35:26.282 INFO:tasks.workunit.client.0.smithi046.stderr:57926: instructing pg 1.4 on osd.3 to scrub 2024-09-18T14:35:26.282 INFO:tasks.workunit.client.0.smithi046.stderr:57926: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:26.282 INFO:tasks.workunit.client.0.smithi046.stderr:57926: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:26.283 INFO:tasks.workunit.client.0.smithi046.stderr:57926: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.4") | .state' 2024-09-18T14:35:26.283 INFO:tasks.workunit.client.0.smithi046.stderr:57926: dumped pgs 2024-09-18T14:35:26.283 INFO:tasks.workunit.client.0.smithi046.stderr:57926: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.4 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:26.283 INFO:tasks.workunit.client.0.smithi046.stderr:57926: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.4 2024-09-18T14:35:26.283 INFO:tasks.workunit.client.0.smithi046.stderr:57926: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:26.283 INFO:tasks.workunit.client.0.smithi046.stderr:57926: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:26.283 INFO:tasks.workunit.client.0.smithi046.stderr:57926: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:26.283 INFO:tasks.workunit.client.0.smithi046.stderr:57926: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:26.284 INFO:tasks.workunit.client.0.smithi046.stderr:57926: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:26.284 INFO:tasks.workunit.client.0.smithi046.stderr:57926: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:26.284 INFO:tasks.workunit.client.0.smithi046.stderr:57926: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.4 last_scrub_stamp 2024-09-18T14:35:26.284 INFO:tasks.workunit.client.0.smithi046.stderr:57926: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-09-18T14:35:26.284 INFO:tasks.workunit.client.0.smithi046.stderr:57926: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:26.284 INFO:tasks.workunit.client.0.smithi046.stderr:57926: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-09-18T14:35:26.284 INFO:tasks.workunit.client.0.smithi046.stderr:57926: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:26.284 INFO:tasks.workunit.client.0.smithi046.stderr:57926: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:26.285 INFO:tasks.workunit.client.0.smithi046.stderr:57926: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:26.285 INFO:tasks.workunit.client.0.smithi046.stderr:57926: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:26.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:26.747 INFO:tasks.workunit.client.0.smithi046.stderr:57915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.2 2024-09-18T14:35:26.747 INFO:tasks.workunit.client.0.smithi046.stderr:57915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.2 2024-09-18T14:35:26.747 INFO:tasks.workunit.client.0.smithi046.stderr:57915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-09-18T14:35:26.747 INFO:tasks.workunit.client.0.smithi046.stderr:57915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:26.747 INFO:tasks.workunit.client.0.smithi046.stderr:57915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:26.747 INFO:tasks.workunit.client.0.smithi046.stderr:57915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-09-18T14:35:26.748 INFO:tasks.workunit.client.0.smithi046.stderr:57915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:26.748 INFO:tasks.workunit.client.0.smithi046.stderr:57915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.2 query 2024-09-18T14:35:26.748 INFO:tasks.workunit.client.0.smithi046.stderr:57915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:26.748 INFO:tasks.workunit.client.0.smithi046.stderr:57915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:26.748 INFO:tasks.workunit.client.0.smithi046.stderr:57915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:26.748 INFO:tasks.workunit.client.0.smithi046.stderr:57915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:26.748 INFO:tasks.workunit.client.0.smithi046.stderr:57915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.2 2024-09-18T14:35:26.748 INFO:tasks.workunit.client.0.smithi046.stderr:57915: instructing pg 1.2 on osd.0 to scrub 2024-09-18T14:35:26.748 INFO:tasks.workunit.client.0.smithi046.stderr:57915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:26.748 INFO:tasks.workunit.client.0.smithi046.stderr:57915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.2") | .state' 2024-09-18T14:35:26.748 INFO:tasks.workunit.client.0.smithi046.stderr:57915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:26.748 INFO:tasks.workunit.client.0.smithi046.stderr:57915: dumped pgs 2024-09-18T14:35:26.749 INFO:tasks.workunit.client.0.smithi046.stderr:57915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.2 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:26.749 INFO:tasks.workunit.client.0.smithi046.stderr:57915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.2 2024-09-18T14:35:26.749 INFO:tasks.workunit.client.0.smithi046.stderr:57915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:26.749 INFO:tasks.workunit.client.0.smithi046.stderr:57915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:26.749 INFO:tasks.workunit.client.0.smithi046.stderr:57915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:26.749 INFO:tasks.workunit.client.0.smithi046.stderr:57915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:26.749 INFO:tasks.workunit.client.0.smithi046.stderr:57915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:26.749 INFO:tasks.workunit.client.0.smithi046.stderr:57915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:26.749 INFO:tasks.workunit.client.0.smithi046.stderr:57915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.2 last_scrub_stamp 2024-09-18T14:35:26.749 INFO:tasks.workunit.client.0.smithi046.stderr:57915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-09-18T14:35:26.749 INFO:tasks.workunit.client.0.smithi046.stderr:57915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:26.749 INFO:tasks.workunit.client.0.smithi046.stderr:57915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-09-18T14:35:26.749 INFO:tasks.workunit.client.0.smithi046.stderr:57915: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:26.749 INFO:tasks.workunit.client.0.smithi046.stderr:57915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:26.750 INFO:tasks.workunit.client.0.smithi046.stderr:57915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:26.750 INFO:tasks.workunit.client.0.smithi046.stderr:57915: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:26.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:27.007 INFO:tasks.workunit.client.0.smithi046.stderr:58050: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.16 2024-09-18T14:35:27.007 INFO:tasks.workunit.client.0.smithi046.stderr:58050: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.16 2024-09-18T14:35:27.007 INFO:tasks.workunit.client.0.smithi046.stderr:58050: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2024-09-18T14:35:27.008 INFO:tasks.workunit.client.0.smithi046.stderr:58050: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.008 INFO:tasks.workunit.client.0.smithi046.stderr:58050: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.008 INFO:tasks.workunit.client.0.smithi046.stderr:58050: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-09-18T14:35:27.008 INFO:tasks.workunit.client.0.smithi046.stderr:58050: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.008 INFO:tasks.workunit.client.0.smithi046.stderr:58050: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.16 query 2024-09-18T14:35:27.008 INFO:tasks.workunit.client.0.smithi046.stderr:58050: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:27.008 INFO:tasks.workunit.client.0.smithi046.stderr:58050: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:27.008 INFO:tasks.workunit.client.0.smithi046.stderr:58050: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:27.008 INFO:tasks.workunit.client.0.smithi046.stderr:58050: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:27.008 INFO:tasks.workunit.client.0.smithi046.stderr:58050: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.16 2024-09-18T14:35:27.008 INFO:tasks.workunit.client.0.smithi046.stderr:58050: instructing pg 1.16 on osd.6 to scrub 2024-09-18T14:35:27.008 INFO:tasks.workunit.client.0.smithi046.stderr:58050: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.16") | .state' 2024-09-18T14:35:27.008 INFO:tasks.workunit.client.0.smithi046.stderr:58050: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:27.009 INFO:tasks.workunit.client.0.smithi046.stderr:58050: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:27.009 INFO:tasks.workunit.client.0.smithi046.stderr:58050: dumped pgs 2024-09-18T14:35:27.009 INFO:tasks.workunit.client.0.smithi046.stderr:58050: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.16 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.009 INFO:tasks.workunit.client.0.smithi046.stderr:58050: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.16 2024-09-18T14:35:27.009 INFO:tasks.workunit.client.0.smithi046.stderr:58050: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.009 INFO:tasks.workunit.client.0.smithi046.stderr:58050: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:27.009 INFO:tasks.workunit.client.0.smithi046.stderr:58050: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:27.009 INFO:tasks.workunit.client.0.smithi046.stderr:58050: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:27.009 INFO:tasks.workunit.client.0.smithi046.stderr:58050: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:27.009 INFO:tasks.workunit.client.0.smithi046.stderr:58050: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:27.009 INFO:tasks.workunit.client.0.smithi046.stderr:58050: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.16 last_scrub_stamp 2024-09-18T14:35:27.009 INFO:tasks.workunit.client.0.smithi046.stderr:58050: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2024-09-18T14:35:27.009 INFO:tasks.workunit.client.0.smithi046.stderr:58050: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.010 INFO:tasks.workunit.client.0.smithi046.stderr:58050: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.010 INFO:tasks.workunit.client.0.smithi046.stderr:58050: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-09-18T14:35:27.010 INFO:tasks.workunit.client.0.smithi046.stderr:58050: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:27.010 INFO:tasks.workunit.client.0.smithi046.stderr:58050: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:27.010 INFO:tasks.workunit.client.0.smithi046.stderr:58050: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:27.010 INFO:tasks.workunit.client.0.smithi046.stderr:et_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.010 INFO:tasks.workunit.client.0.smithi046.stderr:58027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:27.010 INFO:tasks.workunit.client.0.smithi046.stderr:58027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:27.010 INFO:tasks.workunit.client.0.smithi046.stderr:58027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test true = true 2024-09-18T14:35:27.010 INFO:tasks.workunit.client.0.smithi046.stderr:58027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:199: pg_scrub_mod: return 2 2024-09-18T14:35:27.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:27.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 2 2024-09-18T14:35:27.058 INFO:tasks.workunit.client.0.smithi046.stderr:57980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.c 2024-09-18T14:35:27.058 INFO:tasks.workunit.client.0.smithi046.stderr:57980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.c 2024-09-18T14:35:27.058 INFO:tasks.workunit.client.0.smithi046.stderr:57980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2024-09-18T14:35:27.058 INFO:tasks.workunit.client.0.smithi046.stderr:57980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.058 INFO:tasks.workunit.client.0.smithi046.stderr:57980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.059 INFO:tasks.workunit.client.0.smithi046.stderr:57980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-09-18T14:35:27.059 INFO:tasks.workunit.client.0.smithi046.stderr:57980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.059 INFO:tasks.workunit.client.0.smithi046.stderr:57980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:27.059 INFO:tasks.workunit.client.0.smithi046.stderr:57980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.c query 2024-09-18T14:35:27.059 INFO:tasks.workunit.client.0.smithi046.stderr:57980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:27.059 INFO:tasks.workunit.client.0.smithi046.stderr:57980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:27.059 INFO:tasks.workunit.client.0.smithi046.stderr:57980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:27.059 INFO:tasks.workunit.client.0.smithi046.stderr:57980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.c 2024-09-18T14:35:27.059 INFO:tasks.workunit.client.0.smithi046.stderr:57980: instructing pg 1.c on osd.3 to scrub 2024-09-18T14:35:27.059 INFO:tasks.workunit.client.0.smithi046.stderr:57980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:27.059 INFO:tasks.workunit.client.0.smithi046.stderr:57980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.c") | .state' 2024-09-18T14:35:27.059 INFO:tasks.workunit.client.0.smithi046.stderr:57980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:27.059 INFO:tasks.workunit.client.0.smithi046.stderr:57980: dumped pgs 2024-09-18T14:35:27.060 INFO:tasks.workunit.client.0.smithi046.stderr:57980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.c 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.060 INFO:tasks.workunit.client.0.smithi046.stderr:57980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.c 2024-09-18T14:35:27.060 INFO:tasks.workunit.client.0.smithi046.stderr:57980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.060 INFO:tasks.workunit.client.0.smithi046.stderr:57980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:27.060 INFO:tasks.workunit.client.0.smithi046.stderr:57980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:27.060 INFO:tasks.workunit.client.0.smithi046.stderr:57980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:27.060 INFO:tasks.workunit.client.0.smithi046.stderr:57980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:27.060 INFO:tasks.workunit.client.0.smithi046.stderr:57980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:27.060 INFO:tasks.workunit.client.0.smithi046.stderr:57980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.c last_scrub_stamp 2024-09-18T14:35:27.060 INFO:tasks.workunit.client.0.smithi046.stderr:57980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2024-09-18T14:35:27.060 INFO:tasks.workunit.client.0.smithi046.stderr:57980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.060 INFO:tasks.workunit.client.0.smithi046.stderr:57980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-09-18T14:35:27.060 INFO:tasks.workunit.client.0.smithi046.stderr:57980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.060 INFO:tasks.workunit.client.0.smithi046.stderr:57980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:27.061 INFO:tasks.workunit.client.0.smithi046.stderr:57980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:27.061 INFO:tasks.workunit.client.0.smithi046.stderr:57980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:27.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:27.068 INFO:tasks.workunit.client.0.smithi046.stderr:57936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.5 2024-09-18T14:35:27.068 INFO:tasks.workunit.client.0.smithi046.stderr:57936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.5 2024-09-18T14:35:27.068 INFO:tasks.workunit.client.0.smithi046.stderr:57936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-09-18T14:35:27.068 INFO:tasks.workunit.client.0.smithi046.stderr:57936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.068 INFO:tasks.workunit.client.0.smithi046.stderr:57936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.068 INFO:tasks.workunit.client.0.smithi046.stderr:57936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-09-18T14:35:27.069 INFO:tasks.workunit.client.0.smithi046.stderr:57936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.069 INFO:tasks.workunit.client.0.smithi046.stderr:57936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.5 query 2024-09-18T14:35:27.069 INFO:tasks.workunit.client.0.smithi046.stderr:57936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:27.069 INFO:tasks.workunit.client.0.smithi046.stderr:57936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:27.069 INFO:tasks.workunit.client.0.smithi046.stderr:57936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:27.069 INFO:tasks.workunit.client.0.smithi046.stderr:57936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:27.069 INFO:tasks.workunit.client.0.smithi046.stderr:57936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.5 2024-09-18T14:35:27.069 INFO:tasks.workunit.client.0.smithi046.stderr:57936: instructing pg 1.5 on osd.4 to scrub 2024-09-18T14:35:27.069 INFO:tasks.workunit.client.0.smithi046.stderr:57936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:27.069 INFO:tasks.workunit.client.0.smithi046.stderr:57936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:27.069 INFO:tasks.workunit.client.0.smithi046.stderr:57936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.5") | .state' 2024-09-18T14:35:27.069 INFO:tasks.workunit.client.0.smithi046.stderr:57936: dumped pgs 2024-09-18T14:35:27.069 INFO:tasks.workunit.client.0.smithi046.stderr:57936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.5 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.069 INFO:tasks.workunit.client.0.smithi046.stderr:57936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.5 2024-09-18T14:35:27.070 INFO:tasks.workunit.client.0.smithi046.stderr:57936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.070 INFO:tasks.workunit.client.0.smithi046.stderr:57936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:27.070 INFO:tasks.workunit.client.0.smithi046.stderr:57936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:27.070 INFO:tasks.workunit.client.0.smithi046.stderr:57936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:27.070 INFO:tasks.workunit.client.0.smithi046.stderr:57936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:27.070 INFO:tasks.workunit.client.0.smithi046.stderr:57936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:27.070 INFO:tasks.workunit.client.0.smithi046.stderr:57936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.5 last_scrub_stamp 2024-09-18T14:35:27.070 INFO:tasks.workunit.client.0.smithi046.stderr:57936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-09-18T14:35:27.070 INFO:tasks.workunit.client.0.smithi046.stderr:57936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.070 INFO:tasks.workunit.client.0.smithi046.stderr:57936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-09-18T14:35:27.070 INFO:tasks.workunit.client.0.smithi046.stderr:57936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.070 INFO:tasks.workunit.client.0.smithi046.stderr:57936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:27.070 INFO:tasks.workunit.client.0.smithi046.stderr:57936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:27.070 INFO:tasks.workunit.client.0.smithi046.stderr:57936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:27.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:27.606 INFO:tasks.workunit.client.0.smithi046.stderr:57997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.e 2024-09-18T14:35:27.606 INFO:tasks.workunit.client.0.smithi046.stderr:57997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.e 2024-09-18T14:35:27.606 INFO:tasks.workunit.client.0.smithi046.stderr:57997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2024-09-18T14:35:27.606 INFO:tasks.workunit.client.0.smithi046.stderr:57997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.606 INFO:tasks.workunit.client.0.smithi046.stderr:57997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.606 INFO:tasks.workunit.client.0.smithi046.stderr:57997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-09-18T14:35:27.606 INFO:tasks.workunit.client.0.smithi046.stderr:57997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.606 INFO:tasks.workunit.client.0.smithi046.stderr:57997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:27.606 INFO:tasks.workunit.client.0.smithi046.stderr:57997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.e query 2024-09-18T14:35:27.606 INFO:tasks.workunit.client.0.smithi046.stderr:57997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:27.607 INFO:tasks.workunit.client.0.smithi046.stderr:57997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:27.607 INFO:tasks.workunit.client.0.smithi046.stderr:57997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:27.607 INFO:tasks.workunit.client.0.smithi046.stderr:57997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.e 2024-09-18T14:35:27.607 INFO:tasks.workunit.client.0.smithi046.stderr:57997: instructing pg 1.e on osd.1 to scrub 2024-09-18T14:35:27.607 INFO:tasks.workunit.client.0.smithi046.stderr:57997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:27.607 INFO:tasks.workunit.client.0.smithi046.stderr:57997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.e") | .state' 2024-09-18T14:35:27.607 INFO:tasks.workunit.client.0.smithi046.stderr:57997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:27.607 INFO:tasks.workunit.client.0.smithi046.stderr:57997: dumped pgs 2024-09-18T14:35:27.607 INFO:tasks.workunit.client.0.smithi046.stderr:57997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.e 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.607 INFO:tasks.workunit.client.0.smithi046.stderr:57997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.e 2024-09-18T14:35:27.607 INFO:tasks.workunit.client.0.smithi046.stderr:57997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.607 INFO:tasks.workunit.client.0.smithi046.stderr:57997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:27.607 INFO:tasks.workunit.client.0.smithi046.stderr:57997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:27.607 INFO:tasks.workunit.client.0.smithi046.stderr:57997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:27.608 INFO:tasks.workunit.client.0.smithi046.stderr:57997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:27.608 INFO:tasks.workunit.client.0.smithi046.stderr:57997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:27.608 INFO:tasks.workunit.client.0.smithi046.stderr:57997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.e last_scrub_stamp 2024-09-18T14:35:27.608 INFO:tasks.workunit.client.0.smithi046.stderr:57997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2024-09-18T14:35:27.608 INFO:tasks.workunit.client.0.smithi046.stderr:57997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.608 INFO:tasks.workunit.client.0.smithi046.stderr:57997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-09-18T14:35:27.608 INFO:tasks.workunit.client.0.smithi046.stderr:57997: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.608 INFO:tasks.workunit.client.0.smithi046.stderr:57997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:27.608 INFO:tasks.workunit.client.0.smithi046.stderr:57997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:27.608 INFO:tasks.workunit.client.0.smithi046.stderr:57997: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:27.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:27.807 INFO:tasks.workunit.client.0.smithi046.stderr:58069: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.19 2024-09-18T14:35:27.807 INFO:tasks.workunit.client.0.smithi046.stderr:58069: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.19 2024-09-18T14:35:27.807 INFO:tasks.workunit.client.0.smithi046.stderr:58069: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2024-09-18T14:35:27.807 INFO:tasks.workunit.client.0.smithi046.stderr:58069: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.807 INFO:tasks.workunit.client.0.smithi046.stderr:58069: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.807 INFO:tasks.workunit.client.0.smithi046.stderr:58069: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-09-18T14:35:27.808 INFO:tasks.workunit.client.0.smithi046.stderr:58069: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.808 INFO:tasks.workunit.client.0.smithi046.stderr:58069: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.19 query 2024-09-18T14:35:27.808 INFO:tasks.workunit.client.0.smithi046.stderr:58069: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:27.808 INFO:tasks.workunit.client.0.smithi046.stderr:58069: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:27.808 INFO:tasks.workunit.client.0.smithi046.stderr:58069: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:27.808 INFO:tasks.workunit.client.0.smithi046.stderr:58069: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:27.808 INFO:tasks.workunit.client.0.smithi046.stderr:58069: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.19 2024-09-18T14:35:27.808 INFO:tasks.workunit.client.0.smithi046.stderr:58069: instructing pg 1.19 on osd.3 to scrub 2024-09-18T14:35:27.808 INFO:tasks.workunit.client.0.smithi046.stderr:58069: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.19") | .state' 2024-09-18T14:35:27.808 INFO:tasks.workunit.client.0.smithi046.stderr:58069: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:27.808 INFO:tasks.workunit.client.0.smithi046.stderr:58069: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:27.808 INFO:tasks.workunit.client.0.smithi046.stderr:58069: dumped pgs 2024-09-18T14:35:27.808 INFO:tasks.workunit.client.0.smithi046.stderr:58069: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.19 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.809 INFO:tasks.workunit.client.0.smithi046.stderr:58069: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.19 2024-09-18T14:35:27.809 INFO:tasks.workunit.client.0.smithi046.stderr:58069: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.809 INFO:tasks.workunit.client.0.smithi046.stderr:58069: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:27.809 INFO:tasks.workunit.client.0.smithi046.stderr:58069: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:27.809 INFO:tasks.workunit.client.0.smithi046.stderr:58069: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:27.809 INFO:tasks.workunit.client.0.smithi046.stderr:58069: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:27.809 INFO:tasks.workunit.client.0.smithi046.stderr:58069: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:27.809 INFO:tasks.workunit.client.0.smithi046.stderr:58069: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.19 last_scrub_stamp 2024-09-18T14:35:27.809 INFO:tasks.workunit.client.0.smithi046.stderr:58069: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2024-09-18T14:35:27.809 INFO:tasks.workunit.client.0.smithi046.stderr:58069: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.809 INFO:tasks.workunit.client.0.smithi046.stderr:58069: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.809 INFO:tasks.workunit.client.0.smithi046.stderr:58069: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-09-18T14:35:27.809 INFO:tasks.workunit.client.0.smithi046.stderr:58069: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:27.809 INFO:tasks.workunit.client.0.smithi046.stderr:58069: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:27.810 INFO:tasks.workunit.client.0.smithi046.stderr:58069: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:27.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:27.820 INFO:tasks.workunit.client.0.smithi046.stderr:58078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.1a 2024-09-18T14:35:27.820 INFO:tasks.workunit.client.0.smithi046.stderr:58078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.1a 2024-09-18T14:35:27.821 INFO:tasks.workunit.client.0.smithi046.stderr:58078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2024-09-18T14:35:27.821 INFO:tasks.workunit.client.0.smithi046.stderr:58078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.821 INFO:tasks.workunit.client.0.smithi046.stderr:58078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-09-18T14:35:27.821 INFO:tasks.workunit.client.0.smithi046.stderr:58078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.821 INFO:tasks.workunit.client.0.smithi046.stderr:58078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.821 INFO:tasks.workunit.client.0.smithi046.stderr:58078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.1a query 2024-09-18T14:35:27.821 INFO:tasks.workunit.client.0.smithi046.stderr:58078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:27.821 INFO:tasks.workunit.client.0.smithi046.stderr:58078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:27.821 INFO:tasks.workunit.client.0.smithi046.stderr:58078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:27.821 INFO:tasks.workunit.client.0.smithi046.stderr:58078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:27.821 INFO:tasks.workunit.client.0.smithi046.stderr:58078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.1a 2024-09-18T14:35:27.821 INFO:tasks.workunit.client.0.smithi046.stderr:58078: instructing pg 1.1a on osd.1 to scrub 2024-09-18T14:35:27.821 INFO:tasks.workunit.client.0.smithi046.stderr:58078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1a") | .state' 2024-09-18T14:35:27.821 INFO:tasks.workunit.client.0.smithi046.stderr:58078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:27.822 INFO:tasks.workunit.client.0.smithi046.stderr:58078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:27.822 INFO:tasks.workunit.client.0.smithi046.stderr:58078: dumped pgs 2024-09-18T14:35:27.822 INFO:tasks.workunit.client.0.smithi046.stderr:58078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.1a 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.822 INFO:tasks.workunit.client.0.smithi046.stderr:58078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.1a 2024-09-18T14:35:27.822 INFO:tasks.workunit.client.0.smithi046.stderr:58078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.822 INFO:tasks.workunit.client.0.smithi046.stderr:58078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:27.822 INFO:tasks.workunit.client.0.smithi046.stderr:58078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:27.822 INFO:tasks.workunit.client.0.smithi046.stderr:58078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:27.822 INFO:tasks.workunit.client.0.smithi046.stderr:58078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:27.822 INFO:tasks.workunit.client.0.smithi046.stderr:58078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:27.822 INFO:tasks.workunit.client.0.smithi046.stderr:58078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.1a last_scrub_stamp 2024-09-18T14:35:27.822 INFO:tasks.workunit.client.0.smithi046.stderr:58078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2024-09-18T14:35:27.822 INFO:tasks.workunit.client.0.smithi046.stderr:58078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.822 INFO:tasks.workunit.client.0.smithi046.stderr:58078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-09-18T14:35:27.823 INFO:tasks.workunit.client.0.smithi046.stderr:58078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.823 INFO:tasks.workunit.client.0.smithi046.stderr:58078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:27.823 INFO:tasks.workunit.client.0.smithi046.stderr:58078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:27.823 INFO:tasks.workunit.client.0.smithi046.stderr:58078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:27.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:27.841 INFO:tasks.workunit.client.0.smithi046.stderr:58021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.11 2024-09-18T14:35:27.842 INFO:tasks.workunit.client.0.smithi046.stderr:58021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.11 2024-09-18T14:35:27.842 INFO:tasks.workunit.client.0.smithi046.stderr:58021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2024-09-18T14:35:27.842 INFO:tasks.workunit.client.0.smithi046.stderr:58021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.842 INFO:tasks.workunit.client.0.smithi046.stderr:58021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.842 INFO:tasks.workunit.client.0.smithi046.stderr:58021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-09-18T14:35:27.842 INFO:tasks.workunit.client.0.smithi046.stderr:58021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.842 INFO:tasks.workunit.client.0.smithi046.stderr:58021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:27.842 INFO:tasks.workunit.client.0.smithi046.stderr:58021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.11 query 2024-09-18T14:35:27.842 INFO:tasks.workunit.client.0.smithi046.stderr:58021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:27.842 INFO:tasks.workunit.client.0.smithi046.stderr:58021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:27.842 INFO:tasks.workunit.client.0.smithi046.stderr:58021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:27.843 INFO:tasks.workunit.client.0.smithi046.stderr:58021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.11 2024-09-18T14:35:27.843 INFO:tasks.workunit.client.0.smithi046.stderr:58021: instructing pg 1.11 on osd.0 to scrub 2024-09-18T14:35:27.843 INFO:tasks.workunit.client.0.smithi046.stderr:58021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:27.843 INFO:tasks.workunit.client.0.smithi046.stderr:58021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.11") | .state' 2024-09-18T14:35:27.843 INFO:tasks.workunit.client.0.smithi046.stderr:58021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:27.843 INFO:tasks.workunit.client.0.smithi046.stderr:58021: dumped pgs 2024-09-18T14:35:27.843 INFO:tasks.workunit.client.0.smithi046.stderr:58021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.11 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.843 INFO:tasks.workunit.client.0.smithi046.stderr:58021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.11 2024-09-18T14:35:27.843 INFO:tasks.workunit.client.0.smithi046.stderr:58021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.843 INFO:tasks.workunit.client.0.smithi046.stderr:58021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:27.843 INFO:tasks.workunit.client.0.smithi046.stderr:58021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:27.843 INFO:tasks.workunit.client.0.smithi046.stderr:58021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:27.843 INFO:tasks.workunit.client.0.smithi046.stderr:58021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:27.843 INFO:tasks.workunit.client.0.smithi046.stderr:58021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:27.844 INFO:tasks.workunit.client.0.smithi046.stderr:58021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.11 last_scrub_stamp 2024-09-18T14:35:27.844 INFO:tasks.workunit.client.0.smithi046.stderr:58021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2024-09-18T14:35:27.844 INFO:tasks.workunit.client.0.smithi046.stderr:58021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.844 INFO:tasks.workunit.client.0.smithi046.stderr:58021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-09-18T14:35:27.844 INFO:tasks.workunit.client.0.smithi046.stderr:58021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.844 INFO:tasks.workunit.client.0.smithi046.stderr:58021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:27.844 INFO:tasks.workunit.client.0.smithi046.stderr:58021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:27.844 INFO:tasks.workunit.client.0.smithi046.stderr:58021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:27.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:27.891 INFO:tasks.workunit.client.0.smithi046.stderr:58095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.1c 2024-09-18T14:35:27.891 INFO:tasks.workunit.client.0.smithi046.stderr:58095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.1c 2024-09-18T14:35:27.891 INFO:tasks.workunit.client.0.smithi046.stderr:58095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2024-09-18T14:35:27.892 INFO:tasks.workunit.client.0.smithi046.stderr:58095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.892 INFO:tasks.workunit.client.0.smithi046.stderr:58095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.892 INFO:tasks.workunit.client.0.smithi046.stderr:58095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-09-18T14:35:27.892 INFO:tasks.workunit.client.0.smithi046.stderr:58095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.892 INFO:tasks.workunit.client.0.smithi046.stderr:58095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.1c query 2024-09-18T14:35:27.892 INFO:tasks.workunit.client.0.smithi046.stderr:58095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:27.892 INFO:tasks.workunit.client.0.smithi046.stderr:58095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:27.892 INFO:tasks.workunit.client.0.smithi046.stderr:58095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:27.892 INFO:tasks.workunit.client.0.smithi046.stderr:58095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:27.892 INFO:tasks.workunit.client.0.smithi046.stderr:58095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.1c 2024-09-18T14:35:27.892 INFO:tasks.workunit.client.0.smithi046.stderr:58095: instructing pg 1.1c on osd.0 to scrub 2024-09-18T14:35:27.892 INFO:tasks.workunit.client.0.smithi046.stderr:58095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1c") | .state' 2024-09-18T14:35:27.892 INFO:tasks.workunit.client.0.smithi046.stderr:58095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:27.893 INFO:tasks.workunit.client.0.smithi046.stderr:58095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:27.893 INFO:tasks.workunit.client.0.smithi046.stderr:58095: dumped pgs 2024-09-18T14:35:27.893 INFO:tasks.workunit.client.0.smithi046.stderr:58095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.1c 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.893 INFO:tasks.workunit.client.0.smithi046.stderr:58095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.1c 2024-09-18T14:35:27.893 INFO:tasks.workunit.client.0.smithi046.stderr:58095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.893 INFO:tasks.workunit.client.0.smithi046.stderr:58095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:27.893 INFO:tasks.workunit.client.0.smithi046.stderr:58095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:27.893 INFO:tasks.workunit.client.0.smithi046.stderr:58095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:27.893 INFO:tasks.workunit.client.0.smithi046.stderr:58095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:27.893 INFO:tasks.workunit.client.0.smithi046.stderr:58095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:27.893 INFO:tasks.workunit.client.0.smithi046.stderr:58095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.1c last_scrub_stamp 2024-09-18T14:35:27.893 INFO:tasks.workunit.client.0.smithi046.stderr:58095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2024-09-18T14:35:27.894 INFO:tasks.workunit.client.0.smithi046.stderr:58095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.894 INFO:tasks.workunit.client.0.smithi046.stderr:58095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-09-18T14:35:27.894 INFO:tasks.workunit.client.0.smithi046.stderr:58095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.894 INFO:tasks.workunit.client.0.smithi046.stderr:58095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:27.894 INFO:tasks.workunit.client.0.smithi046.stderr:58095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:27.894 INFO:tasks.workunit.client.0.smithi046.stderr:58095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:27.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:27.898 INFO:tasks.workunit.client.0.smithi046.stderr:58008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.10 2024-09-18T14:35:27.898 INFO:tasks.workunit.client.0.smithi046.stderr:58008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.10 2024-09-18T14:35:27.898 INFO:tasks.workunit.client.0.smithi046.stderr:58008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2024-09-18T14:35:27.898 INFO:tasks.workunit.client.0.smithi046.stderr:58008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.898 INFO:tasks.workunit.client.0.smithi046.stderr:58008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.898 INFO:tasks.workunit.client.0.smithi046.stderr:58008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-09-18T14:35:27.898 INFO:tasks.workunit.client.0.smithi046.stderr:58008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.899 INFO:tasks.workunit.client.0.smithi046.stderr:58008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.10 query 2024-09-18T14:35:27.899 INFO:tasks.workunit.client.0.smithi046.stderr:58008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:27.899 INFO:tasks.workunit.client.0.smithi046.stderr:58008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:27.899 INFO:tasks.workunit.client.0.smithi046.stderr:58008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:27.899 INFO:tasks.workunit.client.0.smithi046.stderr:58008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:27.899 INFO:tasks.workunit.client.0.smithi046.stderr:58008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.10 2024-09-18T14:35:27.899 INFO:tasks.workunit.client.0.smithi046.stderr:58008: instructing pg 1.10 on osd.5 to scrub 2024-09-18T14:35:27.899 INFO:tasks.workunit.client.0.smithi046.stderr:58008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:27.899 INFO:tasks.workunit.client.0.smithi046.stderr:58008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:27.899 INFO:tasks.workunit.client.0.smithi046.stderr:58008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.10") | .state' 2024-09-18T14:35:27.899 INFO:tasks.workunit.client.0.smithi046.stderr:58008: dumped pgs 2024-09-18T14:35:27.899 INFO:tasks.workunit.client.0.smithi046.stderr:58008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.10 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.899 INFO:tasks.workunit.client.0.smithi046.stderr:58008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.10 2024-09-18T14:35:27.899 INFO:tasks.workunit.client.0.smithi046.stderr:58008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.900 INFO:tasks.workunit.client.0.smithi046.stderr:58008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:27.900 INFO:tasks.workunit.client.0.smithi046.stderr:58008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:27.900 INFO:tasks.workunit.client.0.smithi046.stderr:58008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:27.900 INFO:tasks.workunit.client.0.smithi046.stderr:58008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:27.900 INFO:tasks.workunit.client.0.smithi046.stderr:58008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:27.900 INFO:tasks.workunit.client.0.smithi046.stderr:58008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.10 last_scrub_stamp 2024-09-18T14:35:27.900 INFO:tasks.workunit.client.0.smithi046.stderr:58008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2024-09-18T14:35:27.900 INFO:tasks.workunit.client.0.smithi046.stderr:58008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.900 INFO:tasks.workunit.client.0.smithi046.stderr:58008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.901 INFO:tasks.workunit.client.0.smithi046.stderr:58008: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-09-18T14:35:27.901 INFO:tasks.workunit.client.0.smithi046.stderr:58008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:27.901 INFO:tasks.workunit.client.0.smithi046.stderr:58008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:27.901 INFO:tasks.workunit.client.0.smithi046.stderr:58008: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:27.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:27.916 INFO:tasks.workunit.client.0.smithi046.stderr:57906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.1 2024-09-18T14:35:27.917 INFO:tasks.workunit.client.0.smithi046.stderr:57906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.1 2024-09-18T14:35:27.917 INFO:tasks.workunit.client.0.smithi046.stderr:57906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-09-18T14:35:27.917 INFO:tasks.workunit.client.0.smithi046.stderr:57906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.917 INFO:tasks.workunit.client.0.smithi046.stderr:57906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.917 INFO:tasks.workunit.client.0.smithi046.stderr:57906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-18T14:35:27.917 INFO:tasks.workunit.client.0.smithi046.stderr:57906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.917 INFO:tasks.workunit.client.0.smithi046.stderr:57906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:27.917 INFO:tasks.workunit.client.0.smithi046.stderr:57906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.1 query 2024-09-18T14:35:27.917 INFO:tasks.workunit.client.0.smithi046.stderr:57906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:27.917 INFO:tasks.workunit.client.0.smithi046.stderr:57906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:27.917 INFO:tasks.workunit.client.0.smithi046.stderr:57906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:27.918 INFO:tasks.workunit.client.0.smithi046.stderr:57906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.1 2024-09-18T14:35:27.918 INFO:tasks.workunit.client.0.smithi046.stderr:57906: instructing pg 1.1 on osd.5 to scrub 2024-09-18T14:35:27.918 INFO:tasks.workunit.client.0.smithi046.stderr:57906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:27.918 INFO:tasks.workunit.client.0.smithi046.stderr:57906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:27.918 INFO:tasks.workunit.client.0.smithi046.stderr:57906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1") | .state' 2024-09-18T14:35:27.918 INFO:tasks.workunit.client.0.smithi046.stderr:57906: dumped pgs 2024-09-18T14:35:27.918 INFO:tasks.workunit.client.0.smithi046.stderr:57906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.1 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.918 INFO:tasks.workunit.client.0.smithi046.stderr:57906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.1 2024-09-18T14:35:27.918 INFO:tasks.workunit.client.0.smithi046.stderr:57906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.918 INFO:tasks.workunit.client.0.smithi046.stderr:57906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:27.918 INFO:tasks.workunit.client.0.smithi046.stderr:57906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:27.918 INFO:tasks.workunit.client.0.smithi046.stderr:57906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:27.918 INFO:tasks.workunit.client.0.smithi046.stderr:57906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:27.918 INFO:tasks.workunit.client.0.smithi046.stderr:57906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:27.919 INFO:tasks.workunit.client.0.smithi046.stderr:57906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-09-18T14:35:27.919 INFO:tasks.workunit.client.0.smithi046.stderr:57906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-09-18T14:35:27.919 INFO:tasks.workunit.client.0.smithi046.stderr:57906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.919 INFO:tasks.workunit.client.0.smithi046.stderr:57906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.919 INFO:tasks.workunit.client.0.smithi046.stderr:57906: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-18T14:35:27.919 INFO:tasks.workunit.client.0.smithi046.stderr:57906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:27.919 INFO:tasks.workunit.client.0.smithi046.stderr:57906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:27.919 INFO:tasks.workunit.client.0.smithi046.stderr:57906: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:27.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:27.920 INFO:tasks.workunit.client.0.smithi046.stderr:57944: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.7 2024-09-18T14:35:27.920 INFO:tasks.workunit.client.0.smithi046.stderr:57944: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.7 2024-09-18T14:35:27.920 INFO:tasks.workunit.client.0.smithi046.stderr:57944: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-09-18T14:35:27.920 INFO:tasks.workunit.client.0.smithi046.stderr:57944: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.920 INFO:tasks.workunit.client.0.smithi046.stderr:57944: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.920 INFO:tasks.workunit.client.0.smithi046.stderr:57944: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-09-18T14:35:27.920 INFO:tasks.workunit.client.0.smithi046.stderr:57944: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.920 INFO:tasks.workunit.client.0.smithi046.stderr:57944: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:27.921 INFO:tasks.workunit.client.0.smithi046.stderr:57944: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.7 query 2024-09-18T14:35:27.921 INFO:tasks.workunit.client.0.smithi046.stderr:57944: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:27.921 INFO:tasks.workunit.client.0.smithi046.stderr:57944: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:27.921 INFO:tasks.workunit.client.0.smithi046.stderr:57944: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:27.921 INFO:tasks.workunit.client.0.smithi046.stderr:57944: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.7 2024-09-18T14:35:27.921 INFO:tasks.workunit.client.0.smithi046.stderr:57944: instructing pg 1.7 on osd.6 to scrub 2024-09-18T14:35:27.921 INFO:tasks.workunit.client.0.smithi046.stderr:57944: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.7") | .state' 2024-09-18T14:35:27.921 INFO:tasks.workunit.client.0.smithi046.stderr:57944: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:27.921 INFO:tasks.workunit.client.0.smithi046.stderr:57944: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:27.921 INFO:tasks.workunit.client.0.smithi046.stderr:57944: dumped pgs 2024-09-18T14:35:27.921 INFO:tasks.workunit.client.0.smithi046.stderr:57944: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.7 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.921 INFO:tasks.workunit.client.0.smithi046.stderr:57944: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.7 2024-09-18T14:35:27.921 INFO:tasks.workunit.client.0.smithi046.stderr:57944: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.922 INFO:tasks.workunit.client.0.smithi046.stderr:57944: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:27.922 INFO:tasks.workunit.client.0.smithi046.stderr:57944: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:27.922 INFO:tasks.workunit.client.0.smithi046.stderr:57944: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:27.922 INFO:tasks.workunit.client.0.smithi046.stderr:57944: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:27.922 INFO:tasks.workunit.client.0.smithi046.stderr:57944: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:27.922 INFO:tasks.workunit.client.0.smithi046.stderr:57944: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.7 last_scrub_stamp 2024-09-18T14:35:27.922 INFO:tasks.workunit.client.0.smithi046.stderr:57944: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-09-18T14:35:27.922 INFO:tasks.workunit.client.0.smithi046.stderr:57944: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.922 INFO:tasks.workunit.client.0.smithi046.stderr:57944: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-09-18T14:35:27.922 INFO:tasks.workunit.client.0.smithi046.stderr:57944: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.922 INFO:tasks.workunit.client.0.smithi046.stderr:57944: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:27.922 INFO:tasks.workunit.client.0.smithi046.stderr:57944: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:27.922 INFO:tasks.workunit.client.0.smithi046.stderr:57944: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:27.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:27.927 INFO:tasks.workunit.client.0.smithi046.stderr:58006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.f 2024-09-18T14:35:27.927 INFO:tasks.workunit.client.0.smithi046.stderr:58006: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.f 2024-09-18T14:35:27.927 INFO:tasks.workunit.client.0.smithi046.stderr:58006: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2024-09-18T14:35:27.927 INFO:tasks.workunit.client.0.smithi046.stderr:58006: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.927 INFO:tasks.workunit.client.0.smithi046.stderr:58006: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.927 INFO:tasks.workunit.client.0.smithi046.stderr:58006: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-09-18T14:35:27.927 INFO:tasks.workunit.client.0.smithi046.stderr:58006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.928 INFO:tasks.workunit.client.0.smithi046.stderr:58006: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.f query 2024-09-18T14:35:27.928 INFO:tasks.workunit.client.0.smithi046.stderr:58006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:27.928 INFO:tasks.workunit.client.0.smithi046.stderr:58006: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:27.928 INFO:tasks.workunit.client.0.smithi046.stderr:58006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:27.928 INFO:tasks.workunit.client.0.smithi046.stderr:58006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:27.928 INFO:tasks.workunit.client.0.smithi046.stderr:58006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.f 2024-09-18T14:35:27.928 INFO:tasks.workunit.client.0.smithi046.stderr:58006: instructing pg 1.f on osd.6 to scrub 2024-09-18T14:35:27.928 INFO:tasks.workunit.client.0.smithi046.stderr:58006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:27.928 INFO:tasks.workunit.client.0.smithi046.stderr:58006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:27.928 INFO:tasks.workunit.client.0.smithi046.stderr:58006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.f") | .state' 2024-09-18T14:35:27.928 INFO:tasks.workunit.client.0.smithi046.stderr:58006: dumped pgs 2024-09-18T14:35:27.928 INFO:tasks.workunit.client.0.smithi046.stderr:58006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.f 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.929 INFO:tasks.workunit.client.0.smithi046.stderr:58006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.f 2024-09-18T14:35:27.929 INFO:tasks.workunit.client.0.smithi046.stderr:58006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.929 INFO:tasks.workunit.client.0.smithi046.stderr:58006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:27.929 INFO:tasks.workunit.client.0.smithi046.stderr:58006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:27.929 INFO:tasks.workunit.client.0.smithi046.stderr:58006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:27.929 INFO:tasks.workunit.client.0.smithi046.stderr:58006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:27.929 INFO:tasks.workunit.client.0.smithi046.stderr:58006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:27.929 INFO:tasks.workunit.client.0.smithi046.stderr:58006: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.f last_scrub_stamp 2024-09-18T14:35:27.929 INFO:tasks.workunit.client.0.smithi046.stderr:58006: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2024-09-18T14:35:27.929 INFO:tasks.workunit.client.0.smithi046.stderr:58006: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.929 INFO:tasks.workunit.client.0.smithi046.stderr:58006: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-09-18T14:35:27.929 INFO:tasks.workunit.client.0.smithi046.stderr:58006: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.929 INFO:tasks.workunit.client.0.smithi046.stderr:58006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:27.930 INFO:tasks.workunit.client.0.smithi046.stderr:58006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:27.930 INFO:tasks.workunit.client.0.smithi046.stderr:58006: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:27.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:27.931 INFO:tasks.workunit.client.0.smithi046.stderr:57923: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.3 2024-09-18T14:35:27.931 INFO:tasks.workunit.client.0.smithi046.stderr:57923: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.3 2024-09-18T14:35:27.931 INFO:tasks.workunit.client.0.smithi046.stderr:57923: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T14:35:27.931 INFO:tasks.workunit.client.0.smithi046.stderr:57923: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.931 INFO:tasks.workunit.client.0.smithi046.stderr:57923: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.931 INFO:tasks.workunit.client.0.smithi046.stderr:57923: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T14:35:27.931 INFO:tasks.workunit.client.0.smithi046.stderr:57923: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.931 INFO:tasks.workunit.client.0.smithi046.stderr:57923: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.3 query 2024-09-18T14:35:27.931 INFO:tasks.workunit.client.0.smithi046.stderr:57923: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:27.931 INFO:tasks.workunit.client.0.smithi046.stderr:57923: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:27.931 INFO:tasks.workunit.client.0.smithi046.stderr:57923: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:27.932 INFO:tasks.workunit.client.0.smithi046.stderr:57923: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:27.932 INFO:tasks.workunit.client.0.smithi046.stderr:57923: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.3 2024-09-18T14:35:27.932 INFO:tasks.workunit.client.0.smithi046.stderr:57923: instructing pg 1.3 on osd.4 to scrub 2024-09-18T14:35:27.932 INFO:tasks.workunit.client.0.smithi046.stderr:57923: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.3") | .state' 2024-09-18T14:35:27.932 INFO:tasks.workunit.client.0.smithi046.stderr:57923: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:27.932 INFO:tasks.workunit.client.0.smithi046.stderr:57923: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:27.932 INFO:tasks.workunit.client.0.smithi046.stderr:57923: dumped pgs 2024-09-18T14:35:27.932 INFO:tasks.workunit.client.0.smithi046.stderr:57923: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.3 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.932 INFO:tasks.workunit.client.0.smithi046.stderr:57923: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.3 2024-09-18T14:35:27.932 INFO:tasks.workunit.client.0.smithi046.stderr:57923: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.932 INFO:tasks.workunit.client.0.smithi046.stderr:57923: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:27.932 INFO:tasks.workunit.client.0.smithi046.stderr:57923: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:27.933 INFO:tasks.workunit.client.0.smithi046.stderr:57923: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:27.933 INFO:tasks.workunit.client.0.smithi046.stderr:57923: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:27.933 INFO:tasks.workunit.client.0.smithi046.stderr:57923: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:27.933 INFO:tasks.workunit.client.0.smithi046.stderr:57923: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-18T14:35:27.933 INFO:tasks.workunit.client.0.smithi046.stderr:57923: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T14:35:27.933 INFO:tasks.workunit.client.0.smithi046.stderr:57923: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.933 INFO:tasks.workunit.client.0.smithi046.stderr:57923: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.933 INFO:tasks.workunit.client.0.smithi046.stderr:57923: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T14:35:27.933 INFO:tasks.workunit.client.0.smithi046.stderr:57923: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:27.933 INFO:tasks.workunit.client.0.smithi046.stderr:57923: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:27.933 INFO:tasks.workunit.client.0.smithi046.stderr:57923: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:27.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:27.941 INFO:tasks.workunit.client.0.smithi046.stderr:58047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.15 2024-09-18T14:35:27.941 INFO:tasks.workunit.client.0.smithi046.stderr:58047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.15 2024-09-18T14:35:27.941 INFO:tasks.workunit.client.0.smithi046.stderr:58047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2024-09-18T14:35:27.941 INFO:tasks.workunit.client.0.smithi046.stderr:58047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.942 INFO:tasks.workunit.client.0.smithi046.stderr:58047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.942 INFO:tasks.workunit.client.0.smithi046.stderr:58047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-09-18T14:35:27.942 INFO:tasks.workunit.client.0.smithi046.stderr:58047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.942 INFO:tasks.workunit.client.0.smithi046.stderr:58047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:27.942 INFO:tasks.workunit.client.0.smithi046.stderr:58047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.15 query 2024-09-18T14:35:27.942 INFO:tasks.workunit.client.0.smithi046.stderr:58047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:27.942 INFO:tasks.workunit.client.0.smithi046.stderr:58047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:27.942 INFO:tasks.workunit.client.0.smithi046.stderr:58047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:27.942 INFO:tasks.workunit.client.0.smithi046.stderr:58047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.15 2024-09-18T14:35:27.942 INFO:tasks.workunit.client.0.smithi046.stderr:58047: instructing pg 1.15 on osd.1 to scrub 2024-09-18T14:35:27.942 INFO:tasks.workunit.client.0.smithi046.stderr:58047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.15") | .state' 2024-09-18T14:35:27.942 INFO:tasks.workunit.client.0.smithi046.stderr:58047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:27.942 INFO:tasks.workunit.client.0.smithi046.stderr:58047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:27.942 INFO:tasks.workunit.client.0.smithi046.stderr:58047: dumped pgs 2024-09-18T14:35:27.943 INFO:tasks.workunit.client.0.smithi046.stderr:58047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.15 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.943 INFO:tasks.workunit.client.0.smithi046.stderr:58047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.15 2024-09-18T14:35:27.943 INFO:tasks.workunit.client.0.smithi046.stderr:58047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.943 INFO:tasks.workunit.client.0.smithi046.stderr:58047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:27.943 INFO:tasks.workunit.client.0.smithi046.stderr:58047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:27.943 INFO:tasks.workunit.client.0.smithi046.stderr:58047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:27.943 INFO:tasks.workunit.client.0.smithi046.stderr:58047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:27.943 INFO:tasks.workunit.client.0.smithi046.stderr:58047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:27.943 INFO:tasks.workunit.client.0.smithi046.stderr:58047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.15 last_scrub_stamp 2024-09-18T14:35:27.943 INFO:tasks.workunit.client.0.smithi046.stderr:58047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2024-09-18T14:35:27.943 INFO:tasks.workunit.client.0.smithi046.stderr:58047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.943 INFO:tasks.workunit.client.0.smithi046.stderr:58047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.943 INFO:tasks.workunit.client.0.smithi046.stderr:58047: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-09-18T14:35:27.943 INFO:tasks.workunit.client.0.smithi046.stderr:58047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:27.944 INFO:tasks.workunit.client.0.smithi046.stderr:58047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:27.944 INFO:tasks.workunit.client.0.smithi046.stderr:58047: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:27.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:27.983 INFO:tasks.workunit.client.0.smithi046.stderr:58054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.17 2024-09-18T14:35:27.983 INFO:tasks.workunit.client.0.smithi046.stderr:58054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.17 2024-09-18T14:35:27.983 INFO:tasks.workunit.client.0.smithi046.stderr:58054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2024-09-18T14:35:27.983 INFO:tasks.workunit.client.0.smithi046.stderr:58054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.983 INFO:tasks.workunit.client.0.smithi046.stderr:58054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-09-18T14:35:27.983 INFO:tasks.workunit.client.0.smithi046.stderr:58054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.984 INFO:tasks.workunit.client.0.smithi046.stderr:58054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.984 INFO:tasks.workunit.client.0.smithi046.stderr:58054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.17 query 2024-09-18T14:35:27.984 INFO:tasks.workunit.client.0.smithi046.stderr:58054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:27.984 INFO:tasks.workunit.client.0.smithi046.stderr:58054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:27.984 INFO:tasks.workunit.client.0.smithi046.stderr:58054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:27.984 INFO:tasks.workunit.client.0.smithi046.stderr:58054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:27.984 INFO:tasks.workunit.client.0.smithi046.stderr:58054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.17 2024-09-18T14:35:27.984 INFO:tasks.workunit.client.0.smithi046.stderr:58054: instructing pg 1.17 on osd.0 to scrub 2024-09-18T14:35:27.984 INFO:tasks.workunit.client.0.smithi046.stderr:58054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:27.985 INFO:tasks.workunit.client.0.smithi046.stderr:58054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.17") | .state' 2024-09-18T14:35:27.985 INFO:tasks.workunit.client.0.smithi046.stderr:58054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:27.985 INFO:tasks.workunit.client.0.smithi046.stderr:58054: dumped pgs 2024-09-18T14:35:27.985 INFO:tasks.workunit.client.0.smithi046.stderr:58054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.17 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.985 INFO:tasks.workunit.client.0.smithi046.stderr:58054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.17 2024-09-18T14:35:27.985 INFO:tasks.workunit.client.0.smithi046.stderr:58054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:27.985 INFO:tasks.workunit.client.0.smithi046.stderr:58054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:27.985 INFO:tasks.workunit.client.0.smithi046.stderr:58054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:27.985 INFO:tasks.workunit.client.0.smithi046.stderr:58054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:27.985 INFO:tasks.workunit.client.0.smithi046.stderr:58054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:27.985 INFO:tasks.workunit.client.0.smithi046.stderr:58054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:27.985 INFO:tasks.workunit.client.0.smithi046.stderr:58054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.17 last_scrub_stamp 2024-09-18T14:35:27.985 INFO:tasks.workunit.client.0.smithi046.stderr:58054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2024-09-18T14:35:27.986 INFO:tasks.workunit.client.0.smithi046.stderr:58054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:27.986 INFO:tasks.workunit.client.0.smithi046.stderr:58054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-09-18T14:35:27.986 INFO:tasks.workunit.client.0.smithi046.stderr:58054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:27.986 INFO:tasks.workunit.client.0.smithi046.stderr:58054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:27.986 INFO:tasks.workunit.client.0.smithi046.stderr:58054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:27.986 INFO:tasks.workunit.client.0.smithi046.stderr:58054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:27.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:28.036 INFO:tasks.workunit.client.0.smithi046.stderr:58108: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.1d 2024-09-18T14:35:28.036 INFO:tasks.workunit.client.0.smithi046.stderr:58108: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.1d 2024-09-18T14:35:28.036 INFO:tasks.workunit.client.0.smithi046.stderr:58108: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2024-09-18T14:35:28.036 INFO:tasks.workunit.client.0.smithi046.stderr:58108: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:28.036 INFO:tasks.workunit.client.0.smithi046.stderr:58108: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:28.036 INFO:tasks.workunit.client.0.smithi046.stderr:58108: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-09-18T14:35:28.036 INFO:tasks.workunit.client.0.smithi046.stderr:58108: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.036 INFO:tasks.workunit.client.0.smithi046.stderr:58108: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.1d query 2024-09-18T14:35:28.036 INFO:tasks.workunit.client.0.smithi046.stderr:58108: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:28.036 INFO:tasks.workunit.client.0.smithi046.stderr:58108: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:28.036 INFO:tasks.workunit.client.0.smithi046.stderr:58108: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:28.036 INFO:tasks.workunit.client.0.smithi046.stderr:58108: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:28.037 INFO:tasks.workunit.client.0.smithi046.stderr:58108: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.1d 2024-09-18T14:35:28.037 INFO:tasks.workunit.client.0.smithi046.stderr:58108: instructing pg 1.1d on osd.1 to scrub 2024-09-18T14:35:28.037 INFO:tasks.workunit.client.0.smithi046.stderr:58108: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:28.037 INFO:tasks.workunit.client.0.smithi046.stderr:58108: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:28.037 INFO:tasks.workunit.client.0.smithi046.stderr:58108: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1d") | .state' 2024-09-18T14:35:28.037 INFO:tasks.workunit.client.0.smithi046.stderr:58108: dumped pgs 2024-09-18T14:35:28.037 INFO:tasks.workunit.client.0.smithi046.stderr:58108: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.1d 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.037 INFO:tasks.workunit.client.0.smithi046.stderr:58108: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.1d 2024-09-18T14:35:28.037 INFO:tasks.workunit.client.0.smithi046.stderr:58108: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.037 INFO:tasks.workunit.client.0.smithi046.stderr:58108: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:28.037 INFO:tasks.workunit.client.0.smithi046.stderr:58108: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:28.037 INFO:tasks.workunit.client.0.smithi046.stderr:58108: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:28.037 INFO:tasks.workunit.client.0.smithi046.stderr:58108: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:28.038 INFO:tasks.workunit.client.0.smithi046.stderr:58108: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:28.038 INFO:tasks.workunit.client.0.smithi046.stderr:58108: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.1d last_scrub_stamp 2024-09-18T14:35:28.038 INFO:tasks.workunit.client.0.smithi046.stderr:58108: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2024-09-18T14:35:28.038 INFO:tasks.workunit.client.0.smithi046.stderr:58108: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:28.038 INFO:tasks.workunit.client.0.smithi046.stderr:58108: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-09-18T14:35:28.038 INFO:tasks.workunit.client.0.smithi046.stderr:58108: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:28.038 INFO:tasks.workunit.client.0.smithi046.stderr:58108: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:28.038 INFO:tasks.workunit.client.0.smithi046.stderr:58108: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:28.038 INFO:tasks.workunit.client.0.smithi046.stderr:58108: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:28.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:28.058 INFO:tasks.workunit.client.0.smithi046.stderr:_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:28.059 INFO:tasks.workunit.client.0.smithi046.stderr:57992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:28.059 INFO:tasks.workunit.client.0.smithi046.stderr:57992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:28.059 INFO:tasks.workunit.client.0.smithi046.stderr:57992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test true = true 2024-09-18T14:35:28.059 INFO:tasks.workunit.client.0.smithi046.stderr:57992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:199: pg_scrub_mod: return 2 2024-09-18T14:35:28.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 2 2024-09-18T14:35:28.298 INFO:tasks.workunit.client.0.smithi046.stderr:58093: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.1b 2024-09-18T14:35:28.299 INFO:tasks.workunit.client.0.smithi046.stderr:58093: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.1b 2024-09-18T14:35:28.299 INFO:tasks.workunit.client.0.smithi046.stderr:58093: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2024-09-18T14:35:28.299 INFO:tasks.workunit.client.0.smithi046.stderr:58093: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:28.299 INFO:tasks.workunit.client.0.smithi046.stderr:58093: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-09-18T14:35:28.299 INFO:tasks.workunit.client.0.smithi046.stderr:58093: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:28.299 INFO:tasks.workunit.client.0.smithi046.stderr:58093: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.299 INFO:tasks.workunit.client.0.smithi046.stderr:58093: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.1b query 2024-09-18T14:35:28.299 INFO:tasks.workunit.client.0.smithi046.stderr:58093: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:28.299 INFO:tasks.workunit.client.0.smithi046.stderr:58093: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:28.299 INFO:tasks.workunit.client.0.smithi046.stderr:58093: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:28.299 INFO:tasks.workunit.client.0.smithi046.stderr:58093: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:28.299 INFO:tasks.workunit.client.0.smithi046.stderr:58093: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.1b 2024-09-18T14:35:28.300 INFO:tasks.workunit.client.0.smithi046.stderr:58093: instructing pg 1.1b on osd.0 to scrub 2024-09-18T14:35:28.300 INFO:tasks.workunit.client.0.smithi046.stderr:58093: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:28.300 INFO:tasks.workunit.client.0.smithi046.stderr:58093: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:28.300 INFO:tasks.workunit.client.0.smithi046.stderr:58093: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1b") | .state' 2024-09-18T14:35:28.300 INFO:tasks.workunit.client.0.smithi046.stderr:58093: dumped pgs 2024-09-18T14:35:28.300 INFO:tasks.workunit.client.0.smithi046.stderr:58093: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.1b 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.300 INFO:tasks.workunit.client.0.smithi046.stderr:58093: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.1b 2024-09-18T14:35:28.300 INFO:tasks.workunit.client.0.smithi046.stderr:58093: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.300 INFO:tasks.workunit.client.0.smithi046.stderr:58093: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:28.300 INFO:tasks.workunit.client.0.smithi046.stderr:58093: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:28.300 INFO:tasks.workunit.client.0.smithi046.stderr:58093: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:28.300 INFO:tasks.workunit.client.0.smithi046.stderr:58093: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:28.300 INFO:tasks.workunit.client.0.smithi046.stderr:58093: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:28.301 INFO:tasks.workunit.client.0.smithi046.stderr:58093: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.1b last_scrub_stamp 2024-09-18T14:35:28.301 INFO:tasks.workunit.client.0.smithi046.stderr:58093: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2024-09-18T14:35:28.301 INFO:tasks.workunit.client.0.smithi046.stderr:58093: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:28.301 INFO:tasks.workunit.client.0.smithi046.stderr:58093: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:28.301 INFO:tasks.workunit.client.0.smithi046.stderr:58093: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-09-18T14:35:28.301 INFO:tasks.workunit.client.0.smithi046.stderr:58093: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:28.301 INFO:tasks.workunit.client.0.smithi046.stderr:58093: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:28.301 INFO:tasks.workunit.client.0.smithi046.stderr:58093: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:28.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:28.330 INFO:tasks.workunit.client.0.smithi046.stderr:57900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.0 2024-09-18T14:35:28.330 INFO:tasks.workunit.client.0.smithi046.stderr:57900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.0 2024-09-18T14:35:28.330 INFO:tasks.workunit.client.0.smithi046.stderr:57900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T14:35:28.330 INFO:tasks.workunit.client.0.smithi046.stderr:57900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:28.331 INFO:tasks.workunit.client.0.smithi046.stderr:57900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:28.331 INFO:tasks.workunit.client.0.smithi046.stderr:57900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T14:35:28.331 INFO:tasks.workunit.client.0.smithi046.stderr:57900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.331 INFO:tasks.workunit.client.0.smithi046.stderr:57900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:28.331 INFO:tasks.workunit.client.0.smithi046.stderr:57900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.0 query 2024-09-18T14:35:28.331 INFO:tasks.workunit.client.0.smithi046.stderr:57900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:28.331 INFO:tasks.workunit.client.0.smithi046.stderr:57900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:28.331 INFO:tasks.workunit.client.0.smithi046.stderr:57900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:28.331 INFO:tasks.workunit.client.0.smithi046.stderr:57900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.0 2024-09-18T14:35:28.331 INFO:tasks.workunit.client.0.smithi046.stderr:57900: instructing pg 1.0 on osd.7 to scrub 2024-09-18T14:35:28.331 INFO:tasks.workunit.client.0.smithi046.stderr:57900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:28.331 INFO:tasks.workunit.client.0.smithi046.stderr:57900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:28.332 INFO:tasks.workunit.client.0.smithi046.stderr:57900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.0") | .state' 2024-09-18T14:35:28.332 INFO:tasks.workunit.client.0.smithi046.stderr:57900: dumped pgs 2024-09-18T14:35:28.332 INFO:tasks.workunit.client.0.smithi046.stderr:57900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.0 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.332 INFO:tasks.workunit.client.0.smithi046.stderr:57900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.0 2024-09-18T14:35:28.332 INFO:tasks.workunit.client.0.smithi046.stderr:57900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.332 INFO:tasks.workunit.client.0.smithi046.stderr:57900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:28.332 INFO:tasks.workunit.client.0.smithi046.stderr:57900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:28.332 INFO:tasks.workunit.client.0.smithi046.stderr:57900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:28.333 INFO:tasks.workunit.client.0.smithi046.stderr:57900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:28.333 INFO:tasks.workunit.client.0.smithi046.stderr:57900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:28.333 INFO:tasks.workunit.client.0.smithi046.stderr:57900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T14:35:28.333 INFO:tasks.workunit.client.0.smithi046.stderr:57900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T14:35:28.333 INFO:tasks.workunit.client.0.smithi046.stderr:57900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:28.333 INFO:tasks.workunit.client.0.smithi046.stderr:57900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T14:35:28.333 INFO:tasks.workunit.client.0.smithi046.stderr:57900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:28.333 INFO:tasks.workunit.client.0.smithi046.stderr:57900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:28.333 INFO:tasks.workunit.client.0.smithi046.stderr:57900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:28.333 INFO:tasks.workunit.client.0.smithi046.stderr:57900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:28.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:28.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 57903 2024-09-18T14:35:28.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 57912 2024-09-18T14:35:28.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 57917 2024-09-18T14:35:28.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 57922 2024-09-18T14:35:28.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 57928 2024-09-18T14:35:28.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 57933 2024-09-18T14:35:28.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 57941 2024-09-18T14:35:28.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 57946 2024-09-18T14:35:28.397 INFO:tasks.workunit.client.0.smithi046.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.12 2024-09-18T14:35:28.397 INFO:tasks.workunit.client.0.smithi046.stderr:58023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.12 2024-09-18T14:35:28.397 INFO:tasks.workunit.client.0.smithi046.stderr:58023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2024-09-18T14:35:28.397 INFO:tasks.workunit.client.0.smithi046.stderr:58023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:28.398 INFO:tasks.workunit.client.0.smithi046.stderr:58023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-09-18T14:35:28.398 INFO:tasks.workunit.client.0.smithi046.stderr:58023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:28.398 INFO:tasks.workunit.client.0.smithi046.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.398 INFO:tasks.workunit.client.0.smithi046.stderr:58023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.12 query 2024-09-18T14:35:28.398 INFO:tasks.workunit.client.0.smithi046.stderr:58023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:28.398 INFO:tasks.workunit.client.0.smithi046.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:28.398 INFO:tasks.workunit.client.0.smithi046.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:28.398 INFO:tasks.workunit.client.0.smithi046.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:28.398 INFO:tasks.workunit.client.0.smithi046.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.12 2024-09-18T14:35:28.398 INFO:tasks.workunit.client.0.smithi046.stderr:58023: instructing pg 1.12 on osd.0 to scrub 2024-09-18T14:35:28.399 INFO:tasks.workunit.client.0.smithi046.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.12") | .state' 2024-09-18T14:35:28.399 INFO:tasks.workunit.client.0.smithi046.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:28.399 INFO:tasks.workunit.client.0.smithi046.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:28.399 INFO:tasks.workunit.client.0.smithi046.stderr:58023: dumped pgs 2024-09-18T14:35:28.399 INFO:tasks.workunit.client.0.smithi046.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.12 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.399 INFO:tasks.workunit.client.0.smithi046.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.12 2024-09-18T14:35:28.399 INFO:tasks.workunit.client.0.smithi046.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.399 INFO:tasks.workunit.client.0.smithi046.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:28.399 INFO:tasks.workunit.client.0.smithi046.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:28.399 INFO:tasks.workunit.client.0.smithi046.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:28.399 INFO:tasks.workunit.client.0.smithi046.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:28.399 INFO:tasks.workunit.client.0.smithi046.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:28.400 INFO:tasks.workunit.client.0.smithi046.stderr:58023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.12 last_scrub_stamp 2024-09-18T14:35:28.400 INFO:tasks.workunit.client.0.smithi046.stderr:58023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2024-09-18T14:35:28.400 INFO:tasks.workunit.client.0.smithi046.stderr:58023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:28.400 INFO:tasks.workunit.client.0.smithi046.stderr:58023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:28.400 INFO:tasks.workunit.client.0.smithi046.stderr:58023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-09-18T14:35:28.400 INFO:tasks.workunit.client.0.smithi046.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:28.400 INFO:tasks.workunit.client.0.smithi046.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:28.400 INFO:tasks.workunit.client.0.smithi046.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:28.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:28.460 INFO:tasks.workunit.client.0.smithi046.stderr:58124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.1f 2024-09-18T14:35:28.461 INFO:tasks.workunit.client.0.smithi046.stderr:58124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.1f 2024-09-18T14:35:28.461 INFO:tasks.workunit.client.0.smithi046.stderr:58124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2024-09-18T14:35:28.461 INFO:tasks.workunit.client.0.smithi046.stderr:58124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:28.461 INFO:tasks.workunit.client.0.smithi046.stderr:58124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:28.461 INFO:tasks.workunit.client.0.smithi046.stderr:58124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-09-18T14:35:28.461 INFO:tasks.workunit.client.0.smithi046.stderr:58124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.462 INFO:tasks.workunit.client.0.smithi046.stderr:58124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:28.462 INFO:tasks.workunit.client.0.smithi046.stderr:58124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.1f query 2024-09-18T14:35:28.462 INFO:tasks.workunit.client.0.smithi046.stderr:58124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:28.463 INFO:tasks.workunit.client.0.smithi046.stderr:58124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:28.463 INFO:tasks.workunit.client.0.smithi046.stderr:58124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:28.463 INFO:tasks.workunit.client.0.smithi046.stderr:58124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.1f 2024-09-18T14:35:28.463 INFO:tasks.workunit.client.0.smithi046.stderr:58124: instructing pg 1.1f on osd.5 to scrub 2024-09-18T14:35:28.463 INFO:tasks.workunit.client.0.smithi046.stderr:58124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:28.463 INFO:tasks.workunit.client.0.smithi046.stderr:58124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1f") | .state' 2024-09-18T14:35:28.463 INFO:tasks.workunit.client.0.smithi046.stderr:58124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:28.464 INFO:tasks.workunit.client.0.smithi046.stderr:58124: dumped pgs 2024-09-18T14:35:28.464 INFO:tasks.workunit.client.0.smithi046.stderr:58124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.1f 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.464 INFO:tasks.workunit.client.0.smithi046.stderr:58124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.1f 2024-09-18T14:35:28.464 INFO:tasks.workunit.client.0.smithi046.stderr:58124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.464 INFO:tasks.workunit.client.0.smithi046.stderr:58124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:28.464 INFO:tasks.workunit.client.0.smithi046.stderr:58124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:28.464 INFO:tasks.workunit.client.0.smithi046.stderr:58124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:28.464 INFO:tasks.workunit.client.0.smithi046.stderr:58124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:28.465 INFO:tasks.workunit.client.0.smithi046.stderr:58124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:28.465 INFO:tasks.workunit.client.0.smithi046.stderr:58124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.1f last_scrub_stamp 2024-09-18T14:35:28.465 INFO:tasks.workunit.client.0.smithi046.stderr:58124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2024-09-18T14:35:28.465 INFO:tasks.workunit.client.0.smithi046.stderr:58124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:28.466 INFO:tasks.workunit.client.0.smithi046.stderr:58124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:28.466 INFO:tasks.workunit.client.0.smithi046.stderr:58124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-09-18T14:35:28.466 INFO:tasks.workunit.client.0.smithi046.stderr:58124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:28.466 INFO:tasks.workunit.client.0.smithi046.stderr:58124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:28.466 INFO:tasks.workunit.client.0.smithi046.stderr:58124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:28.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:28.501 INFO:tasks.workunit.client.0.smithi046.stderr:58068: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.18 2024-09-18T14:35:28.501 INFO:tasks.workunit.client.0.smithi046.stderr:58068: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.18 2024-09-18T14:35:28.501 INFO:tasks.workunit.client.0.smithi046.stderr:58068: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2024-09-18T14:35:28.502 INFO:tasks.workunit.client.0.smithi046.stderr:58068: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:28.502 INFO:tasks.workunit.client.0.smithi046.stderr:58068: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-09-18T14:35:28.502 INFO:tasks.workunit.client.0.smithi046.stderr:58068: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:28.502 INFO:tasks.workunit.client.0.smithi046.stderr:58068: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.502 INFO:tasks.workunit.client.0.smithi046.stderr:58068: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.18 query 2024-09-18T14:35:28.502 INFO:tasks.workunit.client.0.smithi046.stderr:58068: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:28.502 INFO:tasks.workunit.client.0.smithi046.stderr:58068: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:28.502 INFO:tasks.workunit.client.0.smithi046.stderr:58068: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:28.502 INFO:tasks.workunit.client.0.smithi046.stderr:58068: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:28.502 INFO:tasks.workunit.client.0.smithi046.stderr:58068: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.18 2024-09-18T14:35:28.502 INFO:tasks.workunit.client.0.smithi046.stderr:58068: instructing pg 1.18 on osd.5 to scrub 2024-09-18T14:35:28.502 INFO:tasks.workunit.client.0.smithi046.stderr:58068: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:28.503 INFO:tasks.workunit.client.0.smithi046.stderr:58068: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:28.503 INFO:tasks.workunit.client.0.smithi046.stderr:58068: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.18") | .state' 2024-09-18T14:35:28.503 INFO:tasks.workunit.client.0.smithi046.stderr:58068: dumped pgs 2024-09-18T14:35:28.503 INFO:tasks.workunit.client.0.smithi046.stderr:58068: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.18 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.503 INFO:tasks.workunit.client.0.smithi046.stderr:58068: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.18 2024-09-18T14:35:28.503 INFO:tasks.workunit.client.0.smithi046.stderr:58068: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.503 INFO:tasks.workunit.client.0.smithi046.stderr:58068: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:28.503 INFO:tasks.workunit.client.0.smithi046.stderr:58068: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:28.503 INFO:tasks.workunit.client.0.smithi046.stderr:58068: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:28.503 INFO:tasks.workunit.client.0.smithi046.stderr:58068: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:28.504 INFO:tasks.workunit.client.0.smithi046.stderr:58068: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:28.504 INFO:tasks.workunit.client.0.smithi046.stderr:58068: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.18 last_scrub_stamp 2024-09-18T14:35:28.504 INFO:tasks.workunit.client.0.smithi046.stderr:58068: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2024-09-18T14:35:28.504 INFO:tasks.workunit.client.0.smithi046.stderr:58068: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:28.504 INFO:tasks.workunit.client.0.smithi046.stderr:58068: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:28.504 INFO:tasks.workunit.client.0.smithi046.stderr:58068: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-09-18T14:35:28.504 INFO:tasks.workunit.client.0.smithi046.stderr:58068: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:35:24.798505+0000 '>' '' 2024-09-18T14:35:28.504 INFO:tasks.workunit.client.0.smithi046.stderr:58068: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:28.505 INFO:tasks.workunit.client.0.smithi046.stderr:58068: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:28.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:28.616 INFO:tasks.workunit.client.0.smithi046.stderr:58033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.14 2024-09-18T14:35:28.616 INFO:tasks.workunit.client.0.smithi046.stderr:58033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.14 2024-09-18T14:35:28.617 INFO:tasks.workunit.client.0.smithi046.stderr:58033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2024-09-18T14:35:28.617 INFO:tasks.workunit.client.0.smithi046.stderr:58033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:28.617 INFO:tasks.workunit.client.0.smithi046.stderr:58033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-09-18T14:35:28.617 INFO:tasks.workunit.client.0.smithi046.stderr:58033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:28.617 INFO:tasks.workunit.client.0.smithi046.stderr:58033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.617 INFO:tasks.workunit.client.0.smithi046.stderr:58033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.14 query 2024-09-18T14:35:28.617 INFO:tasks.workunit.client.0.smithi046.stderr:58033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:28.617 INFO:tasks.workunit.client.0.smithi046.stderr:58033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:28.617 INFO:tasks.workunit.client.0.smithi046.stderr:58033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:28.618 INFO:tasks.workunit.client.0.smithi046.stderr:58033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:28.618 INFO:tasks.workunit.client.0.smithi046.stderr:58033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.14 2024-09-18T14:35:28.618 INFO:tasks.workunit.client.0.smithi046.stderr:58033: instructing pg 1.14 on osd.2 to scrub 2024-09-18T14:35:28.618 INFO:tasks.workunit.client.0.smithi046.stderr:58033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:28.618 INFO:tasks.workunit.client.0.smithi046.stderr:58033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:28.618 INFO:tasks.workunit.client.0.smithi046.stderr:58033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.14") | .state' 2024-09-18T14:35:28.618 INFO:tasks.workunit.client.0.smithi046.stderr:58033: dumped pgs 2024-09-18T14:35:28.618 INFO:tasks.workunit.client.0.smithi046.stderr:58033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.14 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.618 INFO:tasks.workunit.client.0.smithi046.stderr:58033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.14 2024-09-18T14:35:28.618 INFO:tasks.workunit.client.0.smithi046.stderr:58033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.618 INFO:tasks.workunit.client.0.smithi046.stderr:58033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:28.618 INFO:tasks.workunit.client.0.smithi046.stderr:58033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:28.618 INFO:tasks.workunit.client.0.smithi046.stderr:58033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:28.618 INFO:tasks.workunit.client.0.smithi046.stderr:58033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:28.619 INFO:tasks.workunit.client.0.smithi046.stderr:58033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:28.619 INFO:tasks.workunit.client.0.smithi046.stderr:58033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.14 last_scrub_stamp 2024-09-18T14:35:28.619 INFO:tasks.workunit.client.0.smithi046.stderr:58033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2024-09-18T14:35:28.619 INFO:tasks.workunit.client.0.smithi046.stderr:58033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:28.619 INFO:tasks.workunit.client.0.smithi046.stderr:58033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:28.619 INFO:tasks.workunit.client.0.smithi046.stderr:58033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-09-18T14:35:28.619 INFO:tasks.workunit.client.0.smithi046.stderr:58033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:35:25.261295+0000 '>' '' 2024-09-18T14:35:28.619 INFO:tasks.workunit.client.0.smithi046.stderr:58033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:28.620 INFO:tasks.workunit.client.0.smithi046.stderr:58033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:28.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:28.650 INFO:tasks.workunit.client.0.smithi046.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.9 2024-09-18T14:35:28.651 INFO:tasks.workunit.client.0.smithi046.stderr:57966: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.9 2024-09-18T14:35:28.651 INFO:tasks.workunit.client.0.smithi046.stderr:57966: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2024-09-18T14:35:28.651 INFO:tasks.workunit.client.0.smithi046.stderr:57966: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:28.651 INFO:tasks.workunit.client.0.smithi046.stderr:57966: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:28.651 INFO:tasks.workunit.client.0.smithi046.stderr:57966: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-09-18T14:35:28.651 INFO:tasks.workunit.client.0.smithi046.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.651 INFO:tasks.workunit.client.0.smithi046.stderr:57966: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:28.651 INFO:tasks.workunit.client.0.smithi046.stderr:57966: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.9 query 2024-09-18T14:35:28.651 INFO:tasks.workunit.client.0.smithi046.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:28.651 INFO:tasks.workunit.client.0.smithi046.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:28.652 INFO:tasks.workunit.client.0.smithi046.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:28.652 INFO:tasks.workunit.client.0.smithi046.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.9 2024-09-18T14:35:28.652 INFO:tasks.workunit.client.0.smithi046.stderr:57966: instructing pg 1.9 on osd.1 to scrub 2024-09-18T14:35:28.652 INFO:tasks.workunit.client.0.smithi046.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.9") | .state' 2024-09-18T14:35:28.652 INFO:tasks.workunit.client.0.smithi046.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:28.652 INFO:tasks.workunit.client.0.smithi046.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:28.652 INFO:tasks.workunit.client.0.smithi046.stderr:57966: dumped pgs 2024-09-18T14:35:28.652 INFO:tasks.workunit.client.0.smithi046.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.9 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.652 INFO:tasks.workunit.client.0.smithi046.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.9 2024-09-18T14:35:28.652 INFO:tasks.workunit.client.0.smithi046.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.653 INFO:tasks.workunit.client.0.smithi046.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:28.653 INFO:tasks.workunit.client.0.smithi046.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:28.653 INFO:tasks.workunit.client.0.smithi046.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:28.653 INFO:tasks.workunit.client.0.smithi046.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:28.653 INFO:tasks.workunit.client.0.smithi046.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:28.653 INFO:tasks.workunit.client.0.smithi046.stderr:57966: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.9 last_scrub_stamp 2024-09-18T14:35:28.653 INFO:tasks.workunit.client.0.smithi046.stderr:57966: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2024-09-18T14:35:28.653 INFO:tasks.workunit.client.0.smithi046.stderr:57966: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:28.653 INFO:tasks.workunit.client.0.smithi046.stderr:57966: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:28.653 INFO:tasks.workunit.client.0.smithi046.stderr:57966: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-09-18T14:35:28.653 INFO:tasks.workunit.client.0.smithi046.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:28.654 INFO:tasks.workunit.client.0.smithi046.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:28.654 INFO:tasks.workunit.client.0.smithi046.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:28.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:28.689 INFO:tasks.workunit.client.0.smithi046.stderr:58113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.1e 2024-09-18T14:35:28.689 INFO:tasks.workunit.client.0.smithi046.stderr:58113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.1e 2024-09-18T14:35:28.689 INFO:tasks.workunit.client.0.smithi046.stderr:58113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2024-09-18T14:35:28.689 INFO:tasks.workunit.client.0.smithi046.stderr:58113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:28.689 INFO:tasks.workunit.client.0.smithi046.stderr:58113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:28.690 INFO:tasks.workunit.client.0.smithi046.stderr:58113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-09-18T14:35:28.690 INFO:tasks.workunit.client.0.smithi046.stderr:58113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.690 INFO:tasks.workunit.client.0.smithi046.stderr:58113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.1e query 2024-09-18T14:35:28.690 INFO:tasks.workunit.client.0.smithi046.stderr:58113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:28.690 INFO:tasks.workunit.client.0.smithi046.stderr:58113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:28.690 INFO:tasks.workunit.client.0.smithi046.stderr:58113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:28.690 INFO:tasks.workunit.client.0.smithi046.stderr:58113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:28.690 INFO:tasks.workunit.client.0.smithi046.stderr:58113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.1e 2024-09-18T14:35:28.690 INFO:tasks.workunit.client.0.smithi046.stderr:58113: instructing pg 1.1e on osd.3 to scrub 2024-09-18T14:35:28.690 INFO:tasks.workunit.client.0.smithi046.stderr:58113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:28.690 INFO:tasks.workunit.client.0.smithi046.stderr:58113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:28.690 INFO:tasks.workunit.client.0.smithi046.stderr:58113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1e") | .state' 2024-09-18T14:35:28.691 INFO:tasks.workunit.client.0.smithi046.stderr:58113: dumped pgs 2024-09-18T14:35:28.691 INFO:tasks.workunit.client.0.smithi046.stderr:58113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.1e 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.691 INFO:tasks.workunit.client.0.smithi046.stderr:58113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.1e 2024-09-18T14:35:28.691 INFO:tasks.workunit.client.0.smithi046.stderr:58113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.691 INFO:tasks.workunit.client.0.smithi046.stderr:58113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:28.691 INFO:tasks.workunit.client.0.smithi046.stderr:58113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:28.691 INFO:tasks.workunit.client.0.smithi046.stderr:58113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:28.691 INFO:tasks.workunit.client.0.smithi046.stderr:58113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:28.691 INFO:tasks.workunit.client.0.smithi046.stderr:58113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:28.691 INFO:tasks.workunit.client.0.smithi046.stderr:58113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.1e last_scrub_stamp 2024-09-18T14:35:28.692 INFO:tasks.workunit.client.0.smithi046.stderr:58113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2024-09-18T14:35:28.692 INFO:tasks.workunit.client.0.smithi046.stderr:58113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:28.692 INFO:tasks.workunit.client.0.smithi046.stderr:58113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:28.692 INFO:tasks.workunit.client.0.smithi046.stderr:58113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-09-18T14:35:28.692 INFO:tasks.workunit.client.0.smithi046.stderr:58113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:28.692 INFO:tasks.workunit.client.0.smithi046.stderr:58113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:28.692 INFO:tasks.workunit.client.0.smithi046.stderr:58113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:28.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:28.709 INFO:tasks.workunit.client.0.smithi046.stderr:57968: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.a 2024-09-18T14:35:28.709 INFO:tasks.workunit.client.0.smithi046.stderr:57968: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.a 2024-09-18T14:35:28.709 INFO:tasks.workunit.client.0.smithi046.stderr:57968: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2024-09-18T14:35:28.709 INFO:tasks.workunit.client.0.smithi046.stderr:57968: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:28.709 INFO:tasks.workunit.client.0.smithi046.stderr:57968: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-09-18T14:35:28.709 INFO:tasks.workunit.client.0.smithi046.stderr:57968: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:28.709 INFO:tasks.workunit.client.0.smithi046.stderr:57968: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.710 INFO:tasks.workunit.client.0.smithi046.stderr:57968: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.a query 2024-09-18T14:35:28.710 INFO:tasks.workunit.client.0.smithi046.stderr:57968: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:28.710 INFO:tasks.workunit.client.0.smithi046.stderr:57968: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:28.710 INFO:tasks.workunit.client.0.smithi046.stderr:57968: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:28.710 INFO:tasks.workunit.client.0.smithi046.stderr:57968: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:28.710 INFO:tasks.workunit.client.0.smithi046.stderr:57968: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.a 2024-09-18T14:35:28.710 INFO:tasks.workunit.client.0.smithi046.stderr:57968: instructing pg 1.a on osd.1 to scrub 2024-09-18T14:35:28.710 INFO:tasks.workunit.client.0.smithi046.stderr:57968: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:28.710 INFO:tasks.workunit.client.0.smithi046.stderr:57968: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:28.710 INFO:tasks.workunit.client.0.smithi046.stderr:57968: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.a") | .state' 2024-09-18T14:35:28.710 INFO:tasks.workunit.client.0.smithi046.stderr:57968: dumped pgs 2024-09-18T14:35:28.710 INFO:tasks.workunit.client.0.smithi046.stderr:57968: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.a 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.710 INFO:tasks.workunit.client.0.smithi046.stderr:57968: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.a 2024-09-18T14:35:28.710 INFO:tasks.workunit.client.0.smithi046.stderr:57968: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.711 INFO:tasks.workunit.client.0.smithi046.stderr:57968: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:28.711 INFO:tasks.workunit.client.0.smithi046.stderr:57968: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:28.711 INFO:tasks.workunit.client.0.smithi046.stderr:57968: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:28.711 INFO:tasks.workunit.client.0.smithi046.stderr:57968: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:28.711 INFO:tasks.workunit.client.0.smithi046.stderr:57968: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:28.711 INFO:tasks.workunit.client.0.smithi046.stderr:57968: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.a last_scrub_stamp 2024-09-18T14:35:28.711 INFO:tasks.workunit.client.0.smithi046.stderr:57968: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2024-09-18T14:35:28.711 INFO:tasks.workunit.client.0.smithi046.stderr:57968: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:28.711 INFO:tasks.workunit.client.0.smithi046.stderr:57968: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-09-18T14:35:28.711 INFO:tasks.workunit.client.0.smithi046.stderr:57968: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:28.711 INFO:tasks.workunit.client.0.smithi046.stderr:57968: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:28.711 INFO:tasks.workunit.client.0.smithi046.stderr:57968: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:28.711 INFO:tasks.workunit.client.0.smithi046.stderr:57968: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:28.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:28.727 INFO:tasks.workunit.client.0.smithi046.stderr:57973: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.b 2024-09-18T14:35:28.727 INFO:tasks.workunit.client.0.smithi046.stderr:57973: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.b 2024-09-18T14:35:28.727 INFO:tasks.workunit.client.0.smithi046.stderr:57973: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2024-09-18T14:35:28.727 INFO:tasks.workunit.client.0.smithi046.stderr:57973: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:28.727 INFO:tasks.workunit.client.0.smithi046.stderr:57973: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:28.727 INFO:tasks.workunit.client.0.smithi046.stderr:57973: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-09-18T14:35:28.728 INFO:tasks.workunit.client.0.smithi046.stderr:57973: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.728 INFO:tasks.workunit.client.0.smithi046.stderr:57973: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.b query 2024-09-18T14:35:28.728 INFO:tasks.workunit.client.0.smithi046.stderr:57973: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:28.728 INFO:tasks.workunit.client.0.smithi046.stderr:57973: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:28.728 INFO:tasks.workunit.client.0.smithi046.stderr:57973: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:28.728 INFO:tasks.workunit.client.0.smithi046.stderr:57973: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:28.728 INFO:tasks.workunit.client.0.smithi046.stderr:57973: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.b 2024-09-18T14:35:28.728 INFO:tasks.workunit.client.0.smithi046.stderr:57973: instructing pg 1.b on osd.1 to scrub 2024-09-18T14:35:28.728 INFO:tasks.workunit.client.0.smithi046.stderr:57973: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:28.728 INFO:tasks.workunit.client.0.smithi046.stderr:57973: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.b") | .state' 2024-09-18T14:35:28.728 INFO:tasks.workunit.client.0.smithi046.stderr:57973: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:28.728 INFO:tasks.workunit.client.0.smithi046.stderr:57973: dumped pgs 2024-09-18T14:35:28.729 INFO:tasks.workunit.client.0.smithi046.stderr:57973: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.b 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.729 INFO:tasks.workunit.client.0.smithi046.stderr:57973: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.b 2024-09-18T14:35:28.729 INFO:tasks.workunit.client.0.smithi046.stderr:57973: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.729 INFO:tasks.workunit.client.0.smithi046.stderr:57973: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:28.729 INFO:tasks.workunit.client.0.smithi046.stderr:57973: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:28.729 INFO:tasks.workunit.client.0.smithi046.stderr:57973: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:28.729 INFO:tasks.workunit.client.0.smithi046.stderr:57973: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:28.729 INFO:tasks.workunit.client.0.smithi046.stderr:57973: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:28.729 INFO:tasks.workunit.client.0.smithi046.stderr:57973: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.b last_scrub_stamp 2024-09-18T14:35:28.729 INFO:tasks.workunit.client.0.smithi046.stderr:57973: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2024-09-18T14:35:28.729 INFO:tasks.workunit.client.0.smithi046.stderr:57973: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:28.729 INFO:tasks.workunit.client.0.smithi046.stderr:57973: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:28.729 INFO:tasks.workunit.client.0.smithi046.stderr:57973: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-09-18T14:35:28.729 INFO:tasks.workunit.client.0.smithi046.stderr:57973: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:28.730 INFO:tasks.workunit.client.0.smithi046.stderr:57973: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:28.730 INFO:tasks.workunit.client.0.smithi046.stderr:57973: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:28.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:28.745 INFO:tasks.workunit.client.0.smithi046.stderr:57950: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:185: pg_scrub_mod: local pgid=1.8 2024-09-18T14:35:28.745 INFO:tasks.workunit.client.0.smithi046.stderr:57950: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: get_last_scrub_stamp 1.8 2024-09-18T14:35:28.745 INFO:tasks.workunit.client.0.smithi046.stderr:57950: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2024-09-18T14:35:28.745 INFO:tasks.workunit.client.0.smithi046.stderr:57950: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:28.745 INFO:tasks.workunit.client.0.smithi046.stderr:57950: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-09-18T14:35:28.745 INFO:tasks.workunit.client.0.smithi046.stderr:57950: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:28.745 INFO:tasks.workunit.client.0.smithi046.stderr:57950: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:186: pg_scrub_mod: local last_scrub=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.745 INFO:tasks.workunit.client.0.smithi046.stderr:57950: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: bin/ceph pg 1.8 query 2024-09-18T14:35:28.746 INFO:tasks.workunit.client.0.smithi046.stderr:57950: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 188: bin/ceph: No such file or directory 2024-09-18T14:35:28.746 INFO:tasks.workunit.client.0.smithi046.stderr:57950: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: jq '.acting[0]' 2024-09-18T14:35:28.746 INFO:tasks.workunit.client.0.smithi046.stderr:57950: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:188: pg_scrub_mod: local my_primary= 2024-09-18T14:35:28.746 INFO:tasks.workunit.client.0.smithi046.stderr:57950: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local recovery=false 2024-09-18T14:35:28.746 INFO:tasks.workunit.client.0.smithi046.stderr:57950: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: ceph pg scrub 1.8 2024-09-18T14:35:28.746 INFO:tasks.workunit.client.0.smithi046.stderr:57950: instructing pg 1.8 on osd.5 to scrub 2024-09-18T14:35:28.746 INFO:tasks.workunit.client.0.smithi046.stderr:57950: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-18T14:35:28.746 INFO:tasks.workunit.client.0.smithi046.stderr:57950: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: grep -q recovering 2024-09-18T14:35:28.746 INFO:tasks.workunit.client.0.smithi046.stderr:57950: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.8") | .state' 2024-09-18T14:35:28.746 INFO:tasks.workunit.client.0.smithi046.stderr:57950: dumped pgs 2024-09-18T14:35:28.746 INFO:tasks.workunit.client.0.smithi046.stderr:57950: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: wait_for_scrub_mod 1.8 2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.747 INFO:tasks.workunit.client.0.smithi046.stderr:57950: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:150: wait_for_scrub_mod: local pgid=1.8 2024-09-18T14:35:28.747 INFO:tasks.workunit.client.0.smithi046.stderr:57950: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:151: wait_for_scrub_mod: local orig_primary=2024-09-18T14:34:01.722932+0000 2024-09-18T14:35:28.747 INFO:tasks.workunit.client.0.smithi046.stderr:57950: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:152: wait_for_scrub_mod: local last_scrub= 2024-09-18T14:35:28.747 INFO:tasks.workunit.client.0.smithi046.stderr:57950: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:153: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-18T14:35:28.747 INFO:tasks.workunit.client.0.smithi046.stderr:57950: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i=0 )) 2024-09-18T14:35:28.747 INFO:tasks.workunit.client.0.smithi046.stderr:57950: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: (( i < 300 )) 2024-09-18T14:35:28.747 INFO:tasks.workunit.client.0.smithi046.stderr:57950: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: sleep 0.2 2024-09-18T14:35:28.747 INFO:tasks.workunit.client.0.smithi046.stderr:57950: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: get_last_scrub_stamp 1.8 last_scrub_stamp 2024-09-18T14:35:28.747 INFO:tasks.workunit.client.0.smithi046.stderr:57950: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2024-09-18T14:35:28.747 INFO:tasks.workunit.client.0.smithi046.stderr:57950: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:35:28.747 INFO:tasks.workunit.client.0.smithi046.stderr:57950: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-09-18T14:35:28.747 INFO:tasks.workunit.client.0.smithi046.stderr:57950: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:35:28.760 INFO:tasks.workunit.client.0.smithi046.stderr:57950: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: test 2024-09-18T14:34:01.722932+0000 '>' '' 2024-09-18T14:35:28.760 INFO:tasks.workunit.client.0.smithi046.stderr:57950: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:158: wait_for_scrub_mod: return 0 2024-09-18T14:35:28.760 INFO:tasks.workunit.client.0.smithi046.stderr:57950: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:197: pg_scrub_mod: test false = true 2024-09-18T14:35:28.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T14:35:28.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 57956 2024-09-18T14:35:28.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 57959 2024-09-18T14:35:28.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 57969 2024-09-18T14:35:28.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 57972 2024-09-18T14:35:28.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 57976 2024-09-18T14:35:28.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=2 2024-09-18T14:35:28.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 2 -eq 2 2024-09-18T14:35:28.762 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: expr 0 + 1 2024-09-18T14:35:28.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: recov_scrub_count=1 2024-09-18T14:35:28.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 57993 2024-09-18T14:35:28.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 57998 2024-09-18T14:35:28.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 58000 2024-09-18T14:35:28.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 58011 2024-09-18T14:35:28.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 58013 2024-09-18T14:35:28.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 58017 2024-09-18T14:35:28.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=2 2024-09-18T14:35:28.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 2 -eq 2 2024-09-18T14:35:28.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: expr 1 + 1 2024-09-18T14:35:28.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: recov_scrub_count=2 2024-09-18T14:35:28.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 58024 2024-09-18T14:35:28.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 58029 2024-09-18T14:35:28.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 58037 2024-09-18T14:35:28.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 58042 2024-09-18T14:35:28.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 58055 2024-09-18T14:35:28.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 58058 2024-09-18T14:35:28.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 58070 2024-09-18T14:35:28.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 58079 2024-09-18T14:35:28.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 58088 2024-09-18T14:35:28.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 58097 2024-09-18T14:35:28.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 58102 2024-09-18T14:35:28.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:209: wait_background_check: for pid in $pids 2024-09-18T14:35:28.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: wait 58114 2024-09-18T14:35:28.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:211: wait_background_check: retcode=0 2024-09-18T14:35:28.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: test 0 -eq 2 2024-09-18T14:35:28.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: test 0 -ne 0 2024-09-18T14:35:28.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:223: wait_background_check: eval 'pids='\'''\''' 2024-09-18T14:35:28.769 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:223: wait_background_check: pids= 2024-09-18T14:35:28.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:225: wait_background_check: return 0 2024-09-18T14:35:28.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: return_code=0 2024-09-18T14:35:28.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: '[' 0 -ne 0 ']' 2024-09-18T14:35:28.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:293: TEST_recovery_scrub_2: ERRORS=0 2024-09-18T14:35:28.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:294: TEST_recovery_scrub_2: test 2 -eq 0 2024-09-18T14:35:28.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:300: TEST_recovery_scrub_2: find td/osd-recovery-scrub 2024-09-18T14:35:28.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:300: TEST_recovery_scrub_2: grep '[^/]*\.pid' 2024-09-18T14:35:28.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:300: TEST_recovery_scrub_2: pidfile='td/osd-recovery-scrub/osd.0.pid 2024-09-18T14:35:28.770 INFO:tasks.workunit.client.0.smithi046.stderr:td/osd-recovery-scrub/osd.3.pid 2024-09-18T14:35:28.770 INFO:tasks.workunit.client.0.smithi046.stderr:td/osd-recovery-scrub/mon.a.pid 2024-09-18T14:35:28.770 INFO:tasks.workunit.client.0.smithi046.stderr:td/osd-recovery-scrub/osd.2.pid 2024-09-18T14:35:28.770 INFO:tasks.workunit.client.0.smithi046.stderr:td/osd-recovery-scrub/osd.1.pid 2024-09-18T14:35:28.770 INFO:tasks.workunit.client.0.smithi046.stderr:td/osd-recovery-scrub/osd.6.pid 2024-09-18T14:35:28.770 INFO:tasks.workunit.client.0.smithi046.stderr:td/osd-recovery-scrub/osd.5.pid 2024-09-18T14:35:28.770 INFO:tasks.workunit.client.0.smithi046.stderr:td/osd-recovery-scrub/osd.4.pid 2024-09-18T14:35:28.770 INFO:tasks.workunit.client.0.smithi046.stderr:td/osd-recovery-scrub/osd.7.pid 2024-09-18T14:35:28.770 INFO:tasks.workunit.client.0.smithi046.stderr:td/osd-recovery-scrub/mgr.x.pid' 2024-09-18T14:35:28.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:301: TEST_recovery_scrub_2: cat td/osd-recovery-scrub/osd.0.pid td/osd-recovery-scrub/osd.3.pid td/osd-recovery-scrub/mon.a.pid td/osd-recovery-scrub/osd.2.pid td/osd-recovery-scrub/osd.1.pid td/osd-recovery-scrub/osd.6.pid td/osd-recovery-scrub/osd.5.pid td/osd-recovery-scrub/osd.4.pid td/osd-recovery-scrub/osd.7.pid td/osd-recovery-scrub/mgr.x.pid 2024-09-18T14:35:28.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:301: TEST_recovery_scrub_2: pid='42577 2024-09-18T14:35:28.771 INFO:tasks.workunit.client.0.smithi046.stderr:47991 2024-09-18T14:35:28.771 INFO:tasks.workunit.client.0.smithi046.stderr:41510 2024-09-18T14:35:28.771 INFO:tasks.workunit.client.0.smithi046.stderr:46161 2024-09-18T14:35:28.771 INFO:tasks.workunit.client.0.smithi046.stderr:44356 2024-09-18T14:35:28.771 INFO:tasks.workunit.client.0.smithi046.stderr:53403 2024-09-18T14:35:28.771 INFO:tasks.workunit.client.0.smithi046.stderr:51599 2024-09-18T14:35:28.771 INFO:tasks.workunit.client.0.smithi046.stderr:49795 2024-09-18T14:35:28.771 INFO:tasks.workunit.client.0.smithi046.stderr:55208 2024-09-18T14:35:28.771 INFO:tasks.workunit.client.0.smithi046.stderr:41597' 2024-09-18T14:35:28.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: kill -0 42577 47991 41510 46161 44356 53403 51599 49795 55208 41597 2024-09-18T14:35:28.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:310: TEST_recovery_scrub_2: kill_daemons td/osd-recovery-scrub 2024-09-18T14:35:28.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:35:28.772 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:35:28.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:35:28.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:35:28.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:35:31.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:35:31.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:312: TEST_recovery_scrub_2: declare -a err_strings 2024-09-18T14:35:31.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:313: TEST_recovery_scrub_2: err_strings[0]='not scheduling scrubs due to active recovery' 2024-09-18T14:35:31.087 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:315: TEST_recovery_scrub_2: expr 8 - 1 2024-09-18T14:35:31.089 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:315: TEST_recovery_scrub_2: seq 0 7 2024-09-18T14:35:31.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:315: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:35:31.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.0.log 2024-09-18T14:35:31.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:315: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:35:31.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.1.log 2024-09-18T14:35:31.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:315: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:35:31.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.2.log 2024-09-18T14:35:31.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:315: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:35:31.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.3.log 2024-09-18T14:35:31.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:315: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:35:31.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.4.log 2024-09-18T14:35:31.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:315: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:35:31.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.5.log 2024-09-18T14:35:31.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:315: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:35:31.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.6.log 2024-09-18T14:35:31.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:315: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:35:31.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.7.log 2024-09-18T14:35:31.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:319: TEST_recovery_scrub_2: for err_string in "${err_strings[@]}" 2024-09-18T14:35:31.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:321: TEST_recovery_scrub_2: found=false 2024-09-18T14:35:31.146 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:322: TEST_recovery_scrub_2: expr 8 - 1 2024-09-18T14:35:31.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:322: TEST_recovery_scrub_2: seq 0 7 2024-09-18T14:35:31.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:322: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:35:31.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:324: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.0.log 2024-09-18T14:35:31.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:322: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:35:31.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:324: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.1.log 2024-09-18T14:35:31.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:322: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:35:31.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:324: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.2.log 2024-09-18T14:35:31.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:322: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:35:31.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:324: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.3.log 2024-09-18T14:35:31.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:322: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:35:31.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:324: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.4.log 2024-09-18T14:35:31.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:322: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:35:31.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:324: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.5.log 2024-09-18T14:35:31.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:322: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:35:31.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:324: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.6.log 2024-09-18T14:35:31.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:322: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:35:31.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:324: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.7.log 2024-09-18T14:35:31.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:329: TEST_recovery_scrub_2: '[' false = true ']' 2024-09-18T14:35:31.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:335: TEST_recovery_scrub_2: teardown td/osd-recovery-scrub 2024-09-18T14:35:31.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-09-18T14:35:31.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:35:31.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-09-18T14:35:31.194 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:35:31.194 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:35:31.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:35:31.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:35:31.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:35:31.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:35:31.212 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:35:31.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:35:31.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:35:31.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:35:31.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:35:31.215 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:35:31.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:35:31.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:35:31.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:35:31.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:35:31.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:35:31.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:35:31.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-09-18T14:35:31.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:35:31.264 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:35:31.264 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:35:31.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.31256 2024-09-18T14:35:31.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:35:31.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:35:31.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:337: TEST_recovery_scrub_2: '[' 0 '!=' 0 ']' 2024-09-18T14:35:31.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:343: TEST_recovery_scrub_2: echo 'TEST PASSED' 2024-09-18T14:35:31.266 INFO:tasks.workunit.client.0.smithi046.stdout:TEST PASSED 2024-09-18T14:35:31.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:344: TEST_recovery_scrub_2: return 0 2024-09-18T14:35:31.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T14:35:31.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-recovery-scrub 0 2024-09-18T14:35:31.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-09-18T14:35:31.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T14:35:31.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-09-18T14:35:31.267 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:35:31.267 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:35:31.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:35:31.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:35:31.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:35:31.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:35:31.271 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:35:31.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:35:31.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:35:31.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:35:31.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:35:31.276 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:35:31.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:35:31.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:35:31.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:35:31.278 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:35:31.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:35:31.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T14:35:31.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-09-18T14:35:31.283 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:35:31.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:35:31.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.31256 2024-09-18T14:35:31.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.31256 2024-09-18T14:35:31.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:35:31.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:35:31.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T14:35:31.287 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T14:35:31.287 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T14:35:31.301 INFO:tasks.workunit:Running workunit scrub/osd-scrub-dump.sh... 2024-09-18T14:35:31.301 DEBUG:teuthology.orchestra.run.smithi046:workunit test scrub/osd-scrub-dump.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh 2024-09-18T14:35:31.318 INFO:tasks.workunit.client.0.smithi046.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T14:35:31.333 INFO:tasks.workunit.client.0.smithi046.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T14:35:31.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-18T14:35:31.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-18T14:35:31.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T14:35:31.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T14:35:31.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T14:35:31.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T14:35:31.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T14:35:31.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T14:35:31.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-scrub-dump 2024-09-18T14:35:31.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:25: run: local dir=td/osd-scrub-dump 2024-09-18T14:35:31.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:26: run: shift 2024-09-18T14:35:31.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:27: run: local CHUNK_MAX=5 2024-09-18T14:35:31.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:29: run: export CEPH_MON=127.0.0.1:7184 2024-09-18T14:35:31.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:29: run: CEPH_MON=127.0.0.1:7184 2024-09-18T14:35:31.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:30: run: export CEPH_ARGS 2024-09-18T14:35:31.335 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:31: run: uuidgen 2024-09-18T14:35:31.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:31: run: CEPH_ARGS+='--fsid=62e35ea3-143c-47e6-a329-70e4566cc840 --auth-supported=none ' 2024-09-18T14:35:31.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:32: run: CEPH_ARGS+='--mon-host=127.0.0.1:7184 ' 2024-09-18T14:35:31.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:33: run: CEPH_ARGS+='--osd_max_scrubs=4 ' 2024-09-18T14:35:31.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:34: run: CEPH_ARGS+='--osd_scrub_chunk_max=5 ' 2024-09-18T14:35:31.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:35: run: CEPH_ARGS+='--osd_scrub_sleep=2 ' 2024-09-18T14:35:31.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:36: run: CEPH_ARGS+='--osd_pool_default_size=3 ' 2024-09-18T14:35:31.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:40: run: CEPH_ARGS+='--osd_op_queue=wpq ' 2024-09-18T14:35:31.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:42: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-09-18T14:35:31.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:43: run: set 2024-09-18T14:35:31.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T14:35:31.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:43: run: local funcs=TEST_recover_unexpected 2024-09-18T14:35:31.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:44: run: for func in $funcs 2024-09-18T14:35:31.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:45: run: setup td/osd-scrub-dump 2024-09-18T14:35:31.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-dump 2024-09-18T14:35:31.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-dump 2024-09-18T14:35:31.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-dump 2024-09-18T14:35:31.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:35:31.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-dump KILL 2024-09-18T14:35:31.342 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:35:31.342 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:35:31.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:35:31.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:35:31.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:35:31.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:35:31.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:35:31.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:35:31.347 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:35:31.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:35:31.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:35:31.349 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:35:31.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:35:31.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:35:31.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:35:31.351 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:35:31.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:35:31.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:35:31.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-dump 2024-09-18T14:35:31.356 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:35:31.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:35:31.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:35:31.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61461 2024-09-18T14:35:31.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:35:31.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:35:31.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-dump 2024-09-18T14:35:31.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:35:31.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:35:31.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:35:31.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61461 2024-09-18T14:35:31.362 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:35:31.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T14:35:31.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T14:35:31.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:35:31.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-dump 1' TERM HUP INT 2024-09-18T14:35:31.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:46: run: TEST_recover_unexpected td/osd-scrub-dump 2024-09-18T14:35:31.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:52: TEST_recover_unexpected: local dir=td/osd-scrub-dump 2024-09-18T14:35:31.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:53: TEST_recover_unexpected: shift 2024-09-18T14:35:31.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:54: TEST_recover_unexpected: local OSDS=6 2024-09-18T14:35:31.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:55: TEST_recover_unexpected: local PGS=16 2024-09-18T14:35:31.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:56: TEST_recover_unexpected: local POOLS=3 2024-09-18T14:35:31.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:57: TEST_recover_unexpected: local OBJS=1000 2024-09-18T14:35:31.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:59: TEST_recover_unexpected: run_mon td/osd-scrub-dump a 2024-09-18T14:35:31.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-dump 2024-09-18T14:35:31.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:35:31.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:35:31.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:35:31.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-dump/a 2024-09-18T14:35:31.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-dump/a --run-dir=td/osd-scrub-dump 2024-09-18T14:35:31.766 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:35:31.767 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:35:31.767 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:35:31.767 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:35:31.767 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:35:31.767 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:35:31.767 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61461/$cluster-$name.asok' 2024-09-18T14:35:31.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-dump/a '--log-file=td/osd-scrub-dump/$name.log' '--admin-socket=/tmp/ceph-asok.61461/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-dump/log --run-dir=td/osd-scrub-dump '--pid-file=td/osd-scrub-dump/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T14:35:32.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:35:32.015 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:35:32.015 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:35:32.015 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:35:32.015 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:35:32.015 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:35:32.016 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:35:32.016 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:35:32.016 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:35:32.017 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:35:32.017 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:35:32.017 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:35:32.017 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-mon.a.asok 2024-09-18T14:35:32.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:35:32.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61461/ceph-mon.a.asok config get fsid 2024-09-18T14:35:32.109 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:35:32.109 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:35:32.109 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:35:32.109 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:35:32.109 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:35:32.110 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:35:32.110 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:35:32.111 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:35:32.111 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:35:32.111 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:35:32.111 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:35:32.112 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-mon.a.asok 2024-09-18T14:35:32.112 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:35:32.112 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61461/ceph-mon.a.asok config get mon_host 2024-09-18T14:35:32.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:60: TEST_recover_unexpected: run_mgr td/osd-scrub-dump x 2024-09-18T14:35:32.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-dump 2024-09-18T14:35:32.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:35:32.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:35:32.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:35:32.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-dump/x 2024-09-18T14:35:32.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:35:33.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:35:33.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:35:33.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:35:33.073 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:35:33.073 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:35:33.074 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:35:33.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61461/$cluster-$name.asok' 2024-09-18T14:35:33.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:35:33.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-dump/x '--log-file=td/osd-scrub-dump/$name.log' '--admin-socket=/tmp/ceph-asok.61461/$cluster-$name.asok' --run-dir=td/osd-scrub-dump '--pid-file=td/osd-scrub-dump/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:35:33.119 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:35:33.120 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: seq 0 5 2024-09-18T14:35:33.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:35:33.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:63: TEST_recover_unexpected: run_osd td/osd-scrub-dump 0 2024-09-18T14:35:33.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-dump 2024-09-18T14:35:33.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:35:33.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:35:33.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:35:33.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-dump/0 2024-09-18T14:35:33.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=62e35ea3-143c-47e6-a329-70e4566cc840 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd_op_queue=wpq ' 2024-09-18T14:35:33.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:35:33.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:35:33.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:35:33.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-dump/0' 2024-09-18T14:35:33.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-dump/0/journal' 2024-09-18T14:35:33.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:35:33.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:35:33.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-dump' 2024-09-18T14:35:33.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:35:33.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:35:33.127 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:35:33.127 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:35:33.127 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:35:33.127 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:35:33.127 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61461/$cluster-$name.asok' 2024-09-18T14:35:33.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61461/$cluster-$name.asok' 2024-09-18T14:35:33.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:35:33.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:35:33.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:35:33.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-dump/$name.log' 2024-09-18T14:35:33.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-dump/$name.pid' 2024-09-18T14:35:33.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:35:33.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:35:33.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:35:33.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:35:33.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:35:33.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:35:33.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-dump/0 2024-09-18T14:35:33.128 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:35:33.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=af945cf5-6a7a-42e7-a122-e918b3a7b801 2024-09-18T14:35:33.129 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 af945cf5-6a7a-42e7-a122-e918b3a7b801 2024-09-18T14:35:33.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 af945cf5-6a7a-42e7-a122-e918b3a7b801' 2024-09-18T14:35:33.129 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:35:33.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA15epmzPrFCBAAf2OxBKvJG1F8RtWZwHEA0g== 2024-09-18T14:35:33.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA15epmzPrFCBAAf2OxBKvJG1F8RtWZwHEA0g=="}' 2024-09-18T14:35:33.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new af945cf5-6a7a-42e7-a122-e918b3a7b801 -i td/osd-scrub-dump/0/new.json 2024-09-18T14:35:33.441 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:35:33.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-dump/0/new.json 2024-09-18T14:35:33.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=62e35ea3-143c-47e6-a329-70e4566cc840 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd_op_queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/0 --osd-journal=td/osd-scrub-dump/0/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.61461/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA15epmzPrFCBAAf2OxBKvJG1F8RtWZwHEA0g== --osd-uuid af945cf5-6a7a-42e7-a122-e918b3a7b801 2024-09-18T14:35:33.500 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:33.492+0000 7ff9e2d85540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:33.535 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:33.528+0000 7ff9e2d85540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:33.541 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:33.532+0000 7ff9e2d85540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:33.541 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:33.532+0000 7ff9e2d85540 -1 bluestore(td/osd-scrub-dump/0/block) _read_bdev_label failed to open td/osd-scrub-dump/0/block: (2) No such file or directory 2024-09-18T14:35:33.541 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:33.532+0000 7ff9e2d85540 -1 bluestore(td/osd-scrub-dump/0/block) _read_bdev_label failed to open td/osd-scrub-dump/0/block: (2) No such file or directory 2024-09-18T14:35:33.541 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:33.532+0000 7ff9e2d85540 -1 bluestore(td/osd-scrub-dump/0/block) _read_bdev_label failed to open td/osd-scrub-dump/0/block: (2) No such file or directory 2024-09-18T14:35:33.627 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:33.620+0000 7ff9e2d85540 -1 bluestore(td/osd-scrub-dump/0) _read_fsid unparsable uuid 2024-09-18T14:35:39.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-dump/0/keyring 2024-09-18T14:35:39.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:35:39.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:35:39.868 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T14:35:39.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-dump/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:35:40.266 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T14:35:40.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:35:40.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=62e35ea3-143c-47e6-a329-70e4566cc840 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd_op_queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/0 --osd-journal=td/osd-scrub-dump/0/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.61461/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:35:40.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:35:40.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:35:40.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:35:40.303 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:40.296+0000 7f8272f70540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:40.347 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:40.340+0000 7f8272f70540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:40.348 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:40.344+0000 7f8272f70540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:40.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:35:40.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:35:40.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:35:40.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:35:40.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:35:40.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:40.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:35:40.676 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:35:40.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:40.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:35:40.906 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:40.900+0000 7f8272f70540 -1 Falling back to public interface 2024-09-18T14:35:41.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:42.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:42.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:42.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:35:42.034 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:35:42.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:42.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:35:42.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:43.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:43.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:43.364 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:35:43.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:35:43.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:43.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:35:43.588 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:43.583+0000 7f8272f70540 -1 osd.0 0 log_to_monitors true 2024-09-18T14:35:43.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:44.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:44.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:44.722 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:35:44.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:35:44.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:44.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:35:45.092 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/62583,v1:127.0.0.1:6803/62583] [v2:127.0.0.1:6804/62583,v1:127.0.0.1:6805/62583] exists,up af945cf5-6a7a-42e7-a122-e918b3a7b801 2024-09-18T14:35:45.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:35:45.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:35:45.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:35:45.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:35:45.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:63: TEST_recover_unexpected: run_osd td/osd-scrub-dump 1 2024-09-18T14:35:45.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-dump 2024-09-18T14:35:45.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:35:45.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T14:35:45.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:35:45.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-dump/1 2024-09-18T14:35:45.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=62e35ea3-143c-47e6-a329-70e4566cc840 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd_op_queue=wpq ' 2024-09-18T14:35:45.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:35:45.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:35:45.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:35:45.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-dump/1' 2024-09-18T14:35:45.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-dump/1/journal' 2024-09-18T14:35:45.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:35:45.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:35:45.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-dump' 2024-09-18T14:35:45.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:35:45.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:35:45.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:35:45.097 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:35:45.097 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:35:45.097 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:35:45.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61461/$cluster-$name.asok' 2024-09-18T14:35:45.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61461/$cluster-$name.asok' 2024-09-18T14:35:45.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:35:45.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:35:45.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:35:45.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-dump/$name.log' 2024-09-18T14:35:45.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-dump/$name.pid' 2024-09-18T14:35:45.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:35:45.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:35:45.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:35:45.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:35:45.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:35:45.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:35:45.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-dump/1 2024-09-18T14:35:45.100 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:35:45.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=98a08c99-efc6-4a2d-a032-86dc7ad68086 2024-09-18T14:35:45.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 98a08c99-efc6-4a2d-a032-86dc7ad68086' 2024-09-18T14:35:45.100 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 98a08c99-efc6-4a2d-a032-86dc7ad68086 2024-09-18T14:35:45.100 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:35:45.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBB5epmkmpVBxAA6+SEHR5Lo1q2b3W3jgkm7w== 2024-09-18T14:35:45.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBB5epmkmpVBxAA6+SEHR5Lo1q2b3W3jgkm7w=="}' 2024-09-18T14:35:45.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 98a08c99-efc6-4a2d-a032-86dc7ad68086 -i td/osd-scrub-dump/1/new.json 2024-09-18T14:35:45.520 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:35:45.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-dump/1/new.json 2024-09-18T14:35:45.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=62e35ea3-143c-47e6-a329-70e4566cc840 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd_op_queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/1 --osd-journal=td/osd-scrub-dump/1/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.61461/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBB5epmkmpVBxAA6+SEHR5Lo1q2b3W3jgkm7w== --osd-uuid 98a08c99-efc6-4a2d-a032-86dc7ad68086 2024-09-18T14:35:45.581 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:45.575+0000 7fd589280540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:45.583 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:45.575+0000 7fd589280540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:45.585 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:45.579+0000 7fd589280540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:45.586 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:45.579+0000 7fd589280540 -1 bluestore(td/osd-scrub-dump/1/block) _read_bdev_label failed to open td/osd-scrub-dump/1/block: (2) No such file or directory 2024-09-18T14:35:45.586 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:45.579+0000 7fd589280540 -1 bluestore(td/osd-scrub-dump/1/block) _read_bdev_label failed to open td/osd-scrub-dump/1/block: (2) No such file or directory 2024-09-18T14:35:45.586 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:45.579+0000 7fd589280540 -1 bluestore(td/osd-scrub-dump/1/block) _read_bdev_label failed to open td/osd-scrub-dump/1/block: (2) No such file or directory 2024-09-18T14:35:45.651 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:45.643+0000 7fd589280540 -1 bluestore(td/osd-scrub-dump/1) _read_fsid unparsable uuid 2024-09-18T14:35:52.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-dump/1/keyring 2024-09-18T14:35:52.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:35:52.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T14:35:52.024 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T14:35:52.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-dump/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:35:52.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T14:35:52.453 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T14:35:52.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=62e35ea3-143c-47e6-a329-70e4566cc840 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd_op_queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/1 --osd-journal=td/osd-scrub-dump/1/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.61461/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:35:52.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:35:52.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:35:52.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:35:52.493 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:52.487+0000 7f6809134540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:52.500 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:52.495+0000 7f6809134540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:52.501 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:52.495+0000 7f6809134540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:52.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T14:35:52.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:35:52.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:35:52.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:35:52.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:35:52.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:52.771 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:35:52.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:35:52.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:52.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:35:53.051 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:53.043+0000 7f6809134540 -1 Falling back to public interface 2024-09-18T14:35:53.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:54.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:54.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:54.101 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:35:54.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:35:54.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:54.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:35:54.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:55.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:55.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:55.427 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:35:55.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:35:55.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:55.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:35:55.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:56.009 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:56.003+0000 7f6809134540 -1 osd.1 0 log_to_monitors true 2024-09-18T14:35:56.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:56.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:56.756 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:35:56.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:35:56.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:56.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:35:57.135 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/63980,v1:127.0.0.1:6811/63980] [v2:127.0.0.1:6812/63980,v1:127.0.0.1:6813/63980] exists,up 98a08c99-efc6-4a2d-a032-86dc7ad68086 2024-09-18T14:35:57.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:35:57.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:35:57.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:35:57.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:35:57.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:63: TEST_recover_unexpected: run_osd td/osd-scrub-dump 2 2024-09-18T14:35:57.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-dump 2024-09-18T14:35:57.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:35:57.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T14:35:57.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:35:57.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-dump/2 2024-09-18T14:35:57.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=62e35ea3-143c-47e6-a329-70e4566cc840 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd_op_queue=wpq ' 2024-09-18T14:35:57.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:35:57.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:35:57.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:35:57.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-dump/2' 2024-09-18T14:35:57.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-dump/2/journal' 2024-09-18T14:35:57.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:35:57.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:35:57.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-dump' 2024-09-18T14:35:57.139 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:35:57.139 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:35:57.139 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:35:57.141 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:35:57.141 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:35:57.141 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:35:57.141 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61461/$cluster-$name.asok' 2024-09-18T14:35:57.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61461/$cluster-$name.asok' 2024-09-18T14:35:57.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:35:57.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:35:57.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:35:57.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-dump/$name.log' 2024-09-18T14:35:57.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-dump/$name.pid' 2024-09-18T14:35:57.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:35:57.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:35:57.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:35:57.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:35:57.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:35:57.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:35:57.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-dump/2 2024-09-18T14:35:57.143 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:35:57.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=57d17755-49fd-48f7-91ee-a77fc7567095 2024-09-18T14:35:57.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 57d17755-49fd-48f7-91ee-a77fc7567095' 2024-09-18T14:35:57.145 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 57d17755-49fd-48f7-91ee-a77fc7567095 2024-09-18T14:35:57.145 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:35:57.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBN5epmhPX3CRAAWwLbuUsCQMkYc0fX7VadUw== 2024-09-18T14:35:57.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBN5epmhPX3CRAAWwLbuUsCQMkYc0fX7VadUw=="}' 2024-09-18T14:35:57.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 57d17755-49fd-48f7-91ee-a77fc7567095 -i td/osd-scrub-dump/2/new.json 2024-09-18T14:35:57.584 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:35:57.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-dump/2/new.json 2024-09-18T14:35:57.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=62e35ea3-143c-47e6-a329-70e4566cc840 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd_op_queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/2 --osd-journal=td/osd-scrub-dump/2/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.61461/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBN5epmhPX3CRAAWwLbuUsCQMkYc0fX7VadUw== --osd-uuid 57d17755-49fd-48f7-91ee-a77fc7567095 2024-09-18T14:35:57.644 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:57.635+0000 7fbc8802b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:57.648 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:57.643+0000 7fbc8802b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:57.650 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:57.643+0000 7fbc8802b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:57.651 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:57.643+0000 7fbc8802b540 -1 bluestore(td/osd-scrub-dump/2/block) _read_bdev_label failed to open td/osd-scrub-dump/2/block: (2) No such file or directory 2024-09-18T14:35:57.651 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:57.643+0000 7fbc8802b540 -1 bluestore(td/osd-scrub-dump/2/block) _read_bdev_label failed to open td/osd-scrub-dump/2/block: (2) No such file or directory 2024-09-18T14:35:57.651 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:57.643+0000 7fbc8802b540 -1 bluestore(td/osd-scrub-dump/2/block) _read_bdev_label failed to open td/osd-scrub-dump/2/block: (2) No such file or directory 2024-09-18T14:35:57.713 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:35:57.707+0000 7fbc8802b540 -1 bluestore(td/osd-scrub-dump/2) _read_fsid unparsable uuid 2024-09-18T14:36:04.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-dump/2/keyring 2024-09-18T14:36:04.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:36:04.091 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T14:36:04.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T14:36:04.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-dump/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:36:04.493 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T14:36:04.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T14:36:04.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=62e35ea3-143c-47e6-a329-70e4566cc840 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd_op_queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/2 --osd-journal=td/osd-scrub-dump/2/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.61461/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:36:04.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:36:04.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:36:04.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:36:04.532 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:04.523+0000 7f90ef34a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:04.556 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:04.547+0000 7f90ef34a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:04.557 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:04.551+0000 7f90ef34a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:04.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T14:36:04.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:36:04.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:36:04.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:36:04.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:36:04.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:04.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:36:04.849 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:36:04.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:04.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:36:05.111 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:05.103+0000 7f90ef34a540 -1 Falling back to public interface 2024-09-18T14:36:05.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:06.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:06.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:06.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:36:06.222 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:36:06.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:06.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:36:06.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:07.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:07.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:07.553 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:36:07.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:36:07.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:07.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:36:07.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:07.894 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:07.887+0000 7f90ef34a540 -1 osd.2 0 log_to_monitors true 2024-09-18T14:36:08.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:08.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:08.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:36:08.874 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:36:08.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:08.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:36:09.235 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/65381,v1:127.0.0.1:6819/65381] [v2:127.0.0.1:6820/65381,v1:127.0.0.1:6821/65381] exists,up 57d17755-49fd-48f7-91ee-a77fc7567095 2024-09-18T14:36:09.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:36:09.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:36:09.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:36:09.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:36:09.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:63: TEST_recover_unexpected: run_osd td/osd-scrub-dump 3 2024-09-18T14:36:09.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-dump 2024-09-18T14:36:09.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:36:09.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T14:36:09.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:36:09.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-dump/3 2024-09-18T14:36:09.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=62e35ea3-143c-47e6-a329-70e4566cc840 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd_op_queue=wpq ' 2024-09-18T14:36:09.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:36:09.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:36:09.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:36:09.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-dump/3' 2024-09-18T14:36:09.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-dump/3/journal' 2024-09-18T14:36:09.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:36:09.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:36:09.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-dump' 2024-09-18T14:36:09.238 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:36:09.238 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:36:09.238 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:36:09.240 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:36:09.240 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:36:09.240 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:36:09.240 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61461/$cluster-$name.asok' 2024-09-18T14:36:09.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61461/$cluster-$name.asok' 2024-09-18T14:36:09.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:36:09.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:36:09.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:36:09.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-dump/$name.log' 2024-09-18T14:36:09.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-dump/$name.pid' 2024-09-18T14:36:09.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:36:09.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:36:09.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:36:09.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:36:09.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:36:09.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:36:09.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-dump/3 2024-09-18T14:36:09.242 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:36:09.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6e22344c-7a1e-4a53-bf2d-b2d220df9bc5 2024-09-18T14:36:09.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 6e22344c-7a1e-4a53-bf2d-b2d220df9bc5' 2024-09-18T14:36:09.244 INFO:tasks.workunit.client.0.smithi046.stdout:add osd3 6e22344c-7a1e-4a53-bf2d-b2d220df9bc5 2024-09-18T14:36:09.245 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:36:09.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBZ5epmgJzeDxAAdnjmETNWGyNyXZ76udCopA== 2024-09-18T14:36:09.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBZ5epmgJzeDxAAdnjmETNWGyNyXZ76udCopA=="}' 2024-09-18T14:36:09.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6e22344c-7a1e-4a53-bf2d-b2d220df9bc5 -i td/osd-scrub-dump/3/new.json 2024-09-18T14:36:09.731 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:36:09.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-dump/3/new.json 2024-09-18T14:36:09.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=62e35ea3-143c-47e6-a329-70e4566cc840 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd_op_queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/3 --osd-journal=td/osd-scrub-dump/3/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.61461/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBZ5epmgJzeDxAAdnjmETNWGyNyXZ76udCopA== --osd-uuid 6e22344c-7a1e-4a53-bf2d-b2d220df9bc5 2024-09-18T14:36:09.789 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:09.783+0000 7fc224000540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:09.797 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:09.791+0000 7fc224000540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:09.799 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:09.791+0000 7fc224000540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:09.800 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:09.791+0000 7fc224000540 -1 bluestore(td/osd-scrub-dump/3/block) _read_bdev_label failed to open td/osd-scrub-dump/3/block: (2) No such file or directory 2024-09-18T14:36:09.800 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:09.791+0000 7fc224000540 -1 bluestore(td/osd-scrub-dump/3/block) _read_bdev_label failed to open td/osd-scrub-dump/3/block: (2) No such file or directory 2024-09-18T14:36:09.800 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:09.791+0000 7fc224000540 -1 bluestore(td/osd-scrub-dump/3/block) _read_bdev_label failed to open td/osd-scrub-dump/3/block: (2) No such file or directory 2024-09-18T14:36:09.869 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:09.863+0000 7fc224000540 -1 bluestore(td/osd-scrub-dump/3) _read_fsid unparsable uuid 2024-09-18T14:36:16.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-dump/3/keyring 2024-09-18T14:36:16.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:36:16.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T14:36:16.691 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd3 key to auth repository 2024-09-18T14:36:16.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-dump/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:36:17.086 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.3 2024-09-18T14:36:17.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T14:36:17.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=62e35ea3-143c-47e6-a329-70e4566cc840 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd_op_queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/3 --osd-journal=td/osd-scrub-dump/3/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.61461/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:36:17.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:36:17.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:36:17.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:36:17.124 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:17.119+0000 7fc99d36b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:17.160 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:17.151+0000 7fc99d36b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:17.162 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:17.155+0000 7fc99d36b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:17.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T14:36:17.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:36:17.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T14:36:17.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:36:17.437 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:36:17.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:36:17.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:17.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:36:17.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:36:17.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:17.710 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:17.703+0000 7fc99d36b540 -1 Falling back to public interface 2024-09-18T14:36:17.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:18.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:18.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:18.758 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:36:18.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:36:18.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:18.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:36:19.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:20.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:20.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:20.083 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:36:20.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:36:20.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:20.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:36:20.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:20.447 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:20.439+0000 7fc99d36b540 -1 osd.3 0 log_to_monitors true 2024-09-18T14:36:21.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:21.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:21.406 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:36:21.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:36:21.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:21.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:36:21.784 INFO:tasks.workunit.client.0.smithi046.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/66781,v1:127.0.0.1:6827/66781] [v2:127.0.0.1:6828/66781,v1:127.0.0.1:6829/66781] exists,up 6e22344c-7a1e-4a53-bf2d-b2d220df9bc5 2024-09-18T14:36:21.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:36:21.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:36:21.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:36:21.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:36:21.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:63: TEST_recover_unexpected: run_osd td/osd-scrub-dump 4 2024-09-18T14:36:21.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-dump 2024-09-18T14:36:21.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:36:21.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-18T14:36:21.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:36:21.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-dump/4 2024-09-18T14:36:21.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=62e35ea3-143c-47e6-a329-70e4566cc840 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd_op_queue=wpq ' 2024-09-18T14:36:21.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:36:21.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:36:21.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:36:21.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-dump/4' 2024-09-18T14:36:21.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-dump/4/journal' 2024-09-18T14:36:21.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:36:21.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:36:21.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-dump' 2024-09-18T14:36:21.788 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:36:21.788 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:36:21.788 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:36:21.789 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:36:21.789 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:36:21.789 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:36:21.789 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61461/$cluster-$name.asok' 2024-09-18T14:36:21.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61461/$cluster-$name.asok' 2024-09-18T14:36:21.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:36:21.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:36:21.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:36:21.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-dump/$name.log' 2024-09-18T14:36:21.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-dump/$name.pid' 2024-09-18T14:36:21.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:36:21.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:36:21.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:36:21.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:36:21.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:36:21.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:36:21.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-dump/4 2024-09-18T14:36:21.791 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:36:21.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=23617c32-4e53-4b08-8ec9-2bc25a32ac67 2024-09-18T14:36:21.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 23617c32-4e53-4b08-8ec9-2bc25a32ac67' 2024-09-18T14:36:21.792 INFO:tasks.workunit.client.0.smithi046.stdout:add osd4 23617c32-4e53-4b08-8ec9-2bc25a32ac67 2024-09-18T14:36:21.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:36:21.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBl5epmNWWhMBAACgf8NvJCEKY5g1L93l9yuw== 2024-09-18T14:36:21.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBl5epmNWWhMBAACgf8NvJCEKY5g1L93l9yuw=="}' 2024-09-18T14:36:21.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 23617c32-4e53-4b08-8ec9-2bc25a32ac67 -i td/osd-scrub-dump/4/new.json 2024-09-18T14:36:22.226 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:36:22.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-dump/4/new.json 2024-09-18T14:36:22.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=62e35ea3-143c-47e6-a329-70e4566cc840 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd_op_queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/4 --osd-journal=td/osd-scrub-dump/4/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.61461/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBl5epmNWWhMBAACgf8NvJCEKY5g1L93l9yuw== --osd-uuid 23617c32-4e53-4b08-8ec9-2bc25a32ac67 2024-09-18T14:36:22.286 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:22.279+0000 7f3d91f78540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:22.289 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:22.283+0000 7f3d91f78540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:22.290 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:22.283+0000 7f3d91f78540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:22.291 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:22.283+0000 7f3d91f78540 -1 bluestore(td/osd-scrub-dump/4/block) _read_bdev_label failed to open td/osd-scrub-dump/4/block: (2) No such file or directory 2024-09-18T14:36:22.291 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:22.283+0000 7f3d91f78540 -1 bluestore(td/osd-scrub-dump/4/block) _read_bdev_label failed to open td/osd-scrub-dump/4/block: (2) No such file or directory 2024-09-18T14:36:22.292 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:22.283+0000 7f3d91f78540 -1 bluestore(td/osd-scrub-dump/4/block) _read_bdev_label failed to open td/osd-scrub-dump/4/block: (2) No such file or directory 2024-09-18T14:36:22.382 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:22.375+0000 7f3d91f78540 -1 bluestore(td/osd-scrub-dump/4) _read_fsid unparsable uuid 2024-09-18T14:36:29.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-dump/4/keyring 2024-09-18T14:36:29.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:36:29.187 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd4 key to auth repository 2024-09-18T14:36:29.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-18T14:36:29.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-dump/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:36:29.579 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.4 2024-09-18T14:36:29.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-18T14:36:29.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=62e35ea3-143c-47e6-a329-70e4566cc840 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd_op_queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/4 --osd-journal=td/osd-scrub-dump/4/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.61461/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:36:29.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:36:29.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:36:29.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:36:29.616 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:29.611+0000 7fe175368540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:29.650 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:29.643+0000 7fe175368540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:29.652 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:29.643+0000 7fe175368540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:29.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-18T14:36:29.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:36:29.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-18T14:36:29.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:36:29.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:36:29.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:29.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:36:29.930 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:36:29.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:29.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:36:30.199 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:30.191+0000 7fe175368540 -1 Falling back to public interface 2024-09-18T14:36:30.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:31.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:31.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:31.273 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:36:31.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:36:31.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:31.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:36:31.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:32.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:32.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:32.602 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:36:32.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:36:32.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:32.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:36:32.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:32.946 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:32.939+0000 7fe175368540 -1 osd.4 0 log_to_monitors true 2024-09-18T14:36:33.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:33.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:33.942 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:36:33.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:36:33.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:33.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:36:34.350 INFO:tasks.workunit.client.0.smithi046.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/68179,v1:127.0.0.1:6835/68179] [v2:127.0.0.1:6836/68179,v1:127.0.0.1:6837/68179] exists,up 23617c32-4e53-4b08-8ec9-2bc25a32ac67 2024-09-18T14:36:34.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:36:34.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:36:34.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:36:34.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:36:34.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:63: TEST_recover_unexpected: run_osd td/osd-scrub-dump 5 2024-09-18T14:36:34.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-dump 2024-09-18T14:36:34.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:36:34.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-18T14:36:34.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:36:34.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-dump/5 2024-09-18T14:36:34.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=62e35ea3-143c-47e6-a329-70e4566cc840 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd_op_queue=wpq ' 2024-09-18T14:36:34.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:36:34.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:36:34.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:36:34.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-dump/5' 2024-09-18T14:36:34.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-dump/5/journal' 2024-09-18T14:36:34.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:36:34.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:36:34.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-dump' 2024-09-18T14:36:34.354 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:36:34.354 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:36:34.354 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:36:34.356 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:36:34.356 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:36:34.356 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:36:34.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61461/$cluster-$name.asok' 2024-09-18T14:36:34.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61461/$cluster-$name.asok' 2024-09-18T14:36:34.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:36:34.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:36:34.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:36:34.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-dump/$name.log' 2024-09-18T14:36:34.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-dump/$name.pid' 2024-09-18T14:36:34.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:36:34.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:36:34.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:36:34.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:36:34.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:36:34.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:36:34.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-dump/5 2024-09-18T14:36:34.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:36:34.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cb873713-5250-4d7f-915d-98b928036918 2024-09-18T14:36:34.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 cb873713-5250-4d7f-915d-98b928036918' 2024-09-18T14:36:34.360 INFO:tasks.workunit.client.0.smithi046.stdout:add osd5 cb873713-5250-4d7f-915d-98b928036918 2024-09-18T14:36:34.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:36:34.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBy5epmlMPtFhAAJCeTknGX9WRXPMbq5SvZag== 2024-09-18T14:36:34.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBy5epmlMPtFhAAJCeTknGX9WRXPMbq5SvZag=="}' 2024-09-18T14:36:34.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cb873713-5250-4d7f-915d-98b928036918 -i td/osd-scrub-dump/5/new.json 2024-09-18T14:36:34.827 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:36:34.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-dump/5/new.json 2024-09-18T14:36:34.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=62e35ea3-143c-47e6-a329-70e4566cc840 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd_op_queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/5 --osd-journal=td/osd-scrub-dump/5/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.61461/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBy5epmlMPtFhAAJCeTknGX9WRXPMbq5SvZag== --osd-uuid cb873713-5250-4d7f-915d-98b928036918 2024-09-18T14:36:34.889 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:34.883+0000 7efd33a34540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:34.892 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:34.887+0000 7efd33a34540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:34.894 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:34.887+0000 7efd33a34540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:34.895 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:34.887+0000 7efd33a34540 -1 bluestore(td/osd-scrub-dump/5/block) _read_bdev_label failed to open td/osd-scrub-dump/5/block: (2) No such file or directory 2024-09-18T14:36:34.895 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:34.887+0000 7efd33a34540 -1 bluestore(td/osd-scrub-dump/5/block) _read_bdev_label failed to open td/osd-scrub-dump/5/block: (2) No such file or directory 2024-09-18T14:36:34.896 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:34.887+0000 7efd33a34540 -1 bluestore(td/osd-scrub-dump/5/block) _read_bdev_label failed to open td/osd-scrub-dump/5/block: (2) No such file or directory 2024-09-18T14:36:34.948 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:34.939+0000 7efd33a34540 -1 bluestore(td/osd-scrub-dump/5) _read_fsid unparsable uuid 2024-09-18T14:36:41.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-dump/5/keyring 2024-09-18T14:36:41.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:36:41.415 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd5 key to auth repository 2024-09-18T14:36:41.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-18T14:36:41.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-dump/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:36:41.810 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.5 2024-09-18T14:36:41.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-18T14:36:41.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=62e35ea3-143c-47e6-a329-70e4566cc840 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd_op_queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/5 --osd-journal=td/osd-scrub-dump/5/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.61461/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:36:41.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:36:41.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:36:41.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:36:41.848 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:41.839+0000 7fb53ec64540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:41.878 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:41.871+0000 7fb53ec64540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:41.879 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:41.871+0000 7fb53ec64540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:42.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-18T14:36:42.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:36:42.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T14:36:42.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:36:42.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:36:42.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:42.128 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:36:42.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:36:42.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:42.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T14:36:42.423 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:42.415+0000 7fb53ec64540 -1 Falling back to public interface 2024-09-18T14:36:42.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:43.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:43.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:43.466 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:36:43.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:36:43.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:43.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T14:36:43.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:44.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:44.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:44.785 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:36:44.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:36:44.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:44.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T14:36:45.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:45.181 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:36:45.175+0000 7fb53ec64540 -1 osd.5 0 log_to_monitors true 2024-09-18T14:36:46.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:46.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:46.118 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:36:46.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:36:46.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:46.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T14:36:46.518 INFO:tasks.workunit.client.0.smithi046.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/69578,v1:127.0.0.1:6843/69578] [v2:127.0.0.1:6844/69578,v1:127.0.0.1:6845/69578] exists,up cb873713-5250-4d7f-915d-98b928036918 2024-09-18T14:36:46.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:36:46.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:36:46.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:36:46.520 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:66: TEST_recover_unexpected: seq 1 3 2024-09-18T14:36:46.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:66: TEST_recover_unexpected: for i in $(seq 1 $POOLS) 2024-09-18T14:36:46.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:68: TEST_recover_unexpected: create_pool test1 16 16 2024-09-18T14:36:46.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 16 16 2024-09-18T14:36:47.013 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'test1' created 2024-09-18T14:36:47.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:36:48.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:66: TEST_recover_unexpected: for i in $(seq 1 $POOLS) 2024-09-18T14:36:48.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:68: TEST_recover_unexpected: create_pool test2 16 16 2024-09-18T14:36:48.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 16 16 2024-09-18T14:36:49.230 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'test2' created 2024-09-18T14:36:49.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:36:50.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:66: TEST_recover_unexpected: for i in $(seq 1 $POOLS) 2024-09-18T14:36:50.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:68: TEST_recover_unexpected: create_pool test3 16 16 2024-09-18T14:36:50.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 16 16 2024-09-18T14:36:51.603 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'test3' created 2024-09-18T14:36:51.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:36:52.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:71: TEST_recover_unexpected: wait_for_clean 2024-09-18T14:36:52.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:36:52.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:36:52.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:36:52.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T14:36:52.616 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:36:52.617 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:36:52.617 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:36:52.618 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:36:52.618 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:36:52.618 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:36:52.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:36:52.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:36:52.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:36:52.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:36:52.829 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:36:53.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:36:53.319 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:36:53.319 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T14:36:53.320 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T14:36:53.320 INFO:tasks.workunit.client.0.smithi046.stderr:4 2024-09-18T14:36:53.320 INFO:tasks.workunit.client.0.smithi046.stderr:5' 2024-09-18T14:36:53.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:36:53.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:36:53.321 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:36:53.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836495 2024-09-18T14:36:53.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836495 2024-09-18T14:36:53.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836495' 2024-09-18T14:36:53.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:36:53.479 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:36:53.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672973 2024-09-18T14:36:53.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672973 2024-09-18T14:36:53.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836495 1-42949672973' 2024-09-18T14:36:53.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:36:53.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:36:53.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509450 2024-09-18T14:36:53.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509450 2024-09-18T14:36:53.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836495 1-42949672973 2-64424509450' 2024-09-18T14:36:53.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:36:53.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T14:36:54.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345928 2024-09-18T14:36:54.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345928 2024-09-18T14:36:54.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836495 1-42949672973 2-64424509450 3-85899345928' 2024-09-18T14:36:54.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:36:54.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T14:36:54.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182405 2024-09-18T14:36:54.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182405 2024-09-18T14:36:54.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836495 1-42949672973 2-64424509450 3-85899345928 4-107374182405' 2024-09-18T14:36:54.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:36:54.165 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T14:36:54.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018883 2024-09-18T14:36:54.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018883 2024-09-18T14:36:54.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836495 1-42949672973 2-64424509450 3-85899345928 4-107374182405 5-128849018883' 2024-09-18T14:36:54.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:36:54.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836495 2024-09-18T14:36:54.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:36:54.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:36:54.317 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836495 2024-09-18T14:36:54.317 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:36:54.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836495 2024-09-18T14:36:54.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-09-18T14:36:54.319 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836495 2024-09-18T14:36:54.320 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:36:54.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836495 -lt 21474836495 2024-09-18T14:36:54.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:36:54.638 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672973 2024-09-18T14:36:54.638 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:36:54.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:36:54.642 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672973 2024-09-18T14:36:54.642 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:36:54.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672973 2024-09-18T14:36:54.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-09-18T14:36:54.644 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672973 2024-09-18T14:36:54.644 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:36:54.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672973 -lt 42949672973 2024-09-18T14:36:54.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:36:54.963 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509450 2024-09-18T14:36:54.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:36:54.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:36:54.967 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509450 2024-09-18T14:36:54.967 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:36:54.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509450 2024-09-18T14:36:54.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-09-18T14:36:54.970 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509450 2024-09-18T14:36:54.970 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:36:55.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509450 -lt 64424509450 2024-09-18T14:36:55.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:36:55.384 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345928 2024-09-18T14:36:55.384 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:36:55.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T14:36:55.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345928 2024-09-18T14:36:55.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:36:55.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345928 2024-09-18T14:36:55.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345928' 2024-09-18T14:36:55.389 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.3 seq 85899345928 2024-09-18T14:36:55.390 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T14:36:55.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345928 -lt 85899345928 2024-09-18T14:36:55.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:36:55.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182405 2024-09-18T14:36:55.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:36:55.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T14:36:55.724 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182405 2024-09-18T14:36:55.724 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:36:55.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182405 2024-09-18T14:36:55.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182405' 2024-09-18T14:36:55.726 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.4 seq 107374182405 2024-09-18T14:36:55.726 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T14:36:56.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182405 -lt 107374182405 2024-09-18T14:36:56.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:36:56.093 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018883 2024-09-18T14:36:56.093 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:36:56.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T14:36:56.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018883 2024-09-18T14:36:56.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:36:56.098 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.5 seq 128849018883 2024-09-18T14:36:56.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018883 2024-09-18T14:36:56.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018883' 2024-09-18T14:36:56.099 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T14:36:56.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018883 -lt 128849018883 2024-09-18T14:36:56.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:36:56.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:36:56.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:36:56.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 48 == 0 2024-09-18T14:36:56.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:36:56.861 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:36:56.861 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:36:56.861 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:36:56.861 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:36:56.862 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:36:56.862 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:36:57.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=48 2024-09-18T14:36:57.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:36:57.206 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:36:57.206 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:36:57.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 48 = 48 2024-09-18T14:36:57.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:36:57.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:36:57.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:73: TEST_recover_unexpected: dd if=/dev/urandom of=datafile bs=4k count=2 2024-09-18T14:36:57.599 INFO:tasks.workunit.client.0.smithi046.stderr:2+0 records in 2024-09-18T14:36:57.599 INFO:tasks.workunit.client.0.smithi046.stderr:2+0 records out 2024-09-18T14:36:57.599 INFO:tasks.workunit.client.0.smithi046.stderr:8192 bytes (8.2 kB, 8.0 KiB) copied, 0.000280409 s, 29.2 MB/s 2024-09-18T14:36:57.600 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: seq 1 3 2024-09-18T14:36:57.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for i in $(seq 1 $POOLS) 2024-09-18T14:36:57.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: seq 1 1000 2024-09-18T14:36:57.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:36:57.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj1 datafile 2024-09-18T14:36:57.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:36:57.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj2 datafile 2024-09-18T14:36:57.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:36:57.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj3 datafile 2024-09-18T14:36:58.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:36:58.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj4 datafile 2024-09-18T14:36:58.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:36:58.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj5 datafile 2024-09-18T14:36:58.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:36:58.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj6 datafile 2024-09-18T14:36:58.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:36:58.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj7 datafile 2024-09-18T14:36:58.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:36:58.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj8 datafile 2024-09-18T14:36:58.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:36:58.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj9 datafile 2024-09-18T14:36:59.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:36:59.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj10 datafile 2024-09-18T14:36:59.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:36:59.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj11 datafile 2024-09-18T14:36:59.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:36:59.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj12 datafile 2024-09-18T14:36:59.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:36:59.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj13 datafile 2024-09-18T14:36:59.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:36:59.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj14 datafile 2024-09-18T14:36:59.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:36:59.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj15 datafile 2024-09-18T14:37:00.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:00.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj16 datafile 2024-09-18T14:37:00.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:00.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj17 datafile 2024-09-18T14:37:00.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:00.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj18 datafile 2024-09-18T14:37:00.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:00.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj19 datafile 2024-09-18T14:37:00.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:00.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj20 datafile 2024-09-18T14:37:00.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:00.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj21 datafile 2024-09-18T14:37:00.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:00.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj22 datafile 2024-09-18T14:37:01.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:01.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj23 datafile 2024-09-18T14:37:01.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:01.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj24 datafile 2024-09-18T14:37:01.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:01.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj25 datafile 2024-09-18T14:37:01.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:01.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj26 datafile 2024-09-18T14:37:01.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:01.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj27 datafile 2024-09-18T14:37:01.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:01.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj28 datafile 2024-09-18T14:37:02.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:02.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj29 datafile 2024-09-18T14:37:02.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:02.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj30 datafile 2024-09-18T14:37:02.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:02.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj31 datafile 2024-09-18T14:37:02.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:02.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj32 datafile 2024-09-18T14:37:02.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:02.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj33 datafile 2024-09-18T14:37:02.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:02.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj34 datafile 2024-09-18T14:37:02.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:02.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj35 datafile 2024-09-18T14:37:03.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:03.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj36 datafile 2024-09-18T14:37:03.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:03.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj37 datafile 2024-09-18T14:37:03.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:03.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj38 datafile 2024-09-18T14:37:03.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:03.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj39 datafile 2024-09-18T14:37:03.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:03.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj40 datafile 2024-09-18T14:37:03.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:03.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj41 datafile 2024-09-18T14:37:04.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:04.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj42 datafile 2024-09-18T14:37:04.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:04.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj43 datafile 2024-09-18T14:37:04.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:04.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj44 datafile 2024-09-18T14:37:04.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:04.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj45 datafile 2024-09-18T14:37:04.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:04.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj46 datafile 2024-09-18T14:37:04.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:04.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj47 datafile 2024-09-18T14:37:05.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:05.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj48 datafile 2024-09-18T14:37:05.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:05.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj49 datafile 2024-09-18T14:37:05.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:05.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj50 datafile 2024-09-18T14:37:05.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:05.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj51 datafile 2024-09-18T14:37:05.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:05.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj52 datafile 2024-09-18T14:37:05.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:05.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj53 datafile 2024-09-18T14:37:06.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:06.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj54 datafile 2024-09-18T14:37:06.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:06.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj55 datafile 2024-09-18T14:37:06.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:06.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj56 datafile 2024-09-18T14:37:06.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:06.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj57 datafile 2024-09-18T14:37:06.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:06.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj58 datafile 2024-09-18T14:37:06.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:06.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj59 datafile 2024-09-18T14:37:06.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:06.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj60 datafile 2024-09-18T14:37:07.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:07.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj61 datafile 2024-09-18T14:37:07.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:07.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj62 datafile 2024-09-18T14:37:07.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:07.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj63 datafile 2024-09-18T14:37:07.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:07.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj64 datafile 2024-09-18T14:37:07.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:07.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj65 datafile 2024-09-18T14:37:07.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:07.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj66 datafile 2024-09-18T14:37:08.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:08.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj67 datafile 2024-09-18T14:37:08.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:08.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj68 datafile 2024-09-18T14:37:08.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:08.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj69 datafile 2024-09-18T14:37:08.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:08.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj70 datafile 2024-09-18T14:37:08.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:08.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj71 datafile 2024-09-18T14:37:09.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:09.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj72 datafile 2024-09-18T14:37:09.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:09.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj73 datafile 2024-09-18T14:37:09.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:09.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj74 datafile 2024-09-18T14:37:09.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:09.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj75 datafile 2024-09-18T14:37:09.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:09.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj76 datafile 2024-09-18T14:37:09.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:09.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj77 datafile 2024-09-18T14:37:09.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:09.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj78 datafile 2024-09-18T14:37:10.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:10.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj79 datafile 2024-09-18T14:37:10.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:10.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj80 datafile 2024-09-18T14:37:10.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:10.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj81 datafile 2024-09-18T14:37:10.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:10.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj82 datafile 2024-09-18T14:37:10.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:10.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj83 datafile 2024-09-18T14:37:10.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:10.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj84 datafile 2024-09-18T14:37:11.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:11.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj85 datafile 2024-09-18T14:37:11.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:11.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj86 datafile 2024-09-18T14:37:11.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:11.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj87 datafile 2024-09-18T14:37:11.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:11.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj88 datafile 2024-09-18T14:37:11.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:11.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj89 datafile 2024-09-18T14:37:11.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:11.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj90 datafile 2024-09-18T14:37:12.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:12.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj91 datafile 2024-09-18T14:37:12.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:12.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj92 datafile 2024-09-18T14:37:12.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:12.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj93 datafile 2024-09-18T14:37:12.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:12.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj94 datafile 2024-09-18T14:37:12.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:12.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj95 datafile 2024-09-18T14:37:12.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:12.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj96 datafile 2024-09-18T14:37:12.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:12.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj97 datafile 2024-09-18T14:37:13.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:13.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj98 datafile 2024-09-18T14:37:13.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:13.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj99 datafile 2024-09-18T14:37:13.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:13.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj100 datafile 2024-09-18T14:37:13.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:13.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj101 datafile 2024-09-18T14:37:13.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:13.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj102 datafile 2024-09-18T14:37:13.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:13.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj103 datafile 2024-09-18T14:37:14.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:14.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj104 datafile 2024-09-18T14:37:14.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:14.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj105 datafile 2024-09-18T14:37:14.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:14.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj106 datafile 2024-09-18T14:37:14.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:14.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj107 datafile 2024-09-18T14:37:14.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:14.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj108 datafile 2024-09-18T14:37:14.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:14.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj109 datafile 2024-09-18T14:37:15.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:15.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj110 datafile 2024-09-18T14:37:15.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:15.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj111 datafile 2024-09-18T14:37:15.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:15.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj112 datafile 2024-09-18T14:37:15.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:15.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj113 datafile 2024-09-18T14:37:15.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:15.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj114 datafile 2024-09-18T14:37:15.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:15.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj115 datafile 2024-09-18T14:37:15.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:15.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj116 datafile 2024-09-18T14:37:16.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:16.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj117 datafile 2024-09-18T14:37:16.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:16.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj118 datafile 2024-09-18T14:37:16.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:16.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj119 datafile 2024-09-18T14:37:16.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:16.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj120 datafile 2024-09-18T14:37:16.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:16.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj121 datafile 2024-09-18T14:37:16.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:16.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj122 datafile 2024-09-18T14:37:17.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:17.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj123 datafile 2024-09-18T14:37:17.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:17.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj124 datafile 2024-09-18T14:37:17.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:17.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj125 datafile 2024-09-18T14:37:17.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:17.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj126 datafile 2024-09-18T14:37:17.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:17.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj127 datafile 2024-09-18T14:37:17.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:17.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj128 datafile 2024-09-18T14:37:18.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:18.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj129 datafile 2024-09-18T14:37:18.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:18.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj130 datafile 2024-09-18T14:37:18.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:18.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj131 datafile 2024-09-18T14:37:18.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:18.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj132 datafile 2024-09-18T14:37:18.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:18.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj133 datafile 2024-09-18T14:37:18.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:18.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj134 datafile 2024-09-18T14:37:19.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:19.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj135 datafile 2024-09-18T14:37:19.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:19.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj136 datafile 2024-09-18T14:37:19.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:19.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj137 datafile 2024-09-18T14:37:19.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:19.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj138 datafile 2024-09-18T14:37:19.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:19.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj139 datafile 2024-09-18T14:37:19.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:19.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj140 datafile 2024-09-18T14:37:20.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:20.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj141 datafile 2024-09-18T14:37:20.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:20.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj142 datafile 2024-09-18T14:37:20.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:20.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj143 datafile 2024-09-18T14:37:20.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:20.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj144 datafile 2024-09-18T14:37:20.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:20.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj145 datafile 2024-09-18T14:37:20.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:20.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj146 datafile 2024-09-18T14:37:21.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:21.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj147 datafile 2024-09-18T14:37:21.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:21.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj148 datafile 2024-09-18T14:37:21.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:21.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj149 datafile 2024-09-18T14:37:21.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:21.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj150 datafile 2024-09-18T14:37:21.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:21.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj151 datafile 2024-09-18T14:37:21.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:21.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj152 datafile 2024-09-18T14:37:21.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:21.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj153 datafile 2024-09-18T14:37:22.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:22.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj154 datafile 2024-09-18T14:37:22.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:22.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj155 datafile 2024-09-18T14:37:22.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:22.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj156 datafile 2024-09-18T14:37:22.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:22.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj157 datafile 2024-09-18T14:37:22.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:22.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj158 datafile 2024-09-18T14:37:22.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:22.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj159 datafile 2024-09-18T14:37:23.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:23.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj160 datafile 2024-09-18T14:37:23.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:23.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj161 datafile 2024-09-18T14:37:23.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:23.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj162 datafile 2024-09-18T14:37:23.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:23.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj163 datafile 2024-09-18T14:37:23.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:23.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj164 datafile 2024-09-18T14:37:23.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:23.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj165 datafile 2024-09-18T14:37:24.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:24.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj166 datafile 2024-09-18T14:37:24.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:24.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj167 datafile 2024-09-18T14:37:24.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:24.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj168 datafile 2024-09-18T14:37:24.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:24.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj169 datafile 2024-09-18T14:37:24.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:24.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj170 datafile 2024-09-18T14:37:24.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:24.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj171 datafile 2024-09-18T14:37:25.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:25.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj172 datafile 2024-09-18T14:37:25.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:25.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj173 datafile 2024-09-18T14:37:25.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:25.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj174 datafile 2024-09-18T14:37:25.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:25.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj175 datafile 2024-09-18T14:37:25.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:25.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj176 datafile 2024-09-18T14:37:25.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:25.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj177 datafile 2024-09-18T14:37:26.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:26.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj178 datafile 2024-09-18T14:37:26.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:26.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj179 datafile 2024-09-18T14:37:26.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:26.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj180 datafile 2024-09-18T14:37:26.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:26.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj181 datafile 2024-09-18T14:37:26.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:26.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj182 datafile 2024-09-18T14:37:26.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:26.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj183 datafile 2024-09-18T14:37:26.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:26.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj184 datafile 2024-09-18T14:37:27.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:27.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj185 datafile 2024-09-18T14:37:27.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:27.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj186 datafile 2024-09-18T14:37:27.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:27.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj187 datafile 2024-09-18T14:37:27.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:27.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj188 datafile 2024-09-18T14:37:27.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:27.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj189 datafile 2024-09-18T14:37:27.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:27.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj190 datafile 2024-09-18T14:37:28.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:28.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj191 datafile 2024-09-18T14:37:28.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:28.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj192 datafile 2024-09-18T14:37:28.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:28.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj193 datafile 2024-09-18T14:37:28.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:28.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj194 datafile 2024-09-18T14:37:28.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:28.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj195 datafile 2024-09-18T14:37:28.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:28.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj196 datafile 2024-09-18T14:37:29.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:29.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj197 datafile 2024-09-18T14:37:29.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:29.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj198 datafile 2024-09-18T14:37:29.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:29.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj199 datafile 2024-09-18T14:37:29.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:29.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj200 datafile 2024-09-18T14:37:29.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:29.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj201 datafile 2024-09-18T14:37:29.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:29.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj202 datafile 2024-09-18T14:37:29.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:29.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj203 datafile 2024-09-18T14:37:30.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:30.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj204 datafile 2024-09-18T14:37:30.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:30.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj205 datafile 2024-09-18T14:37:30.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:30.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj206 datafile 2024-09-18T14:37:30.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:30.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj207 datafile 2024-09-18T14:37:30.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:30.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj208 datafile 2024-09-18T14:37:30.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:30.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj209 datafile 2024-09-18T14:37:31.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:31.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj210 datafile 2024-09-18T14:37:31.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:31.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj211 datafile 2024-09-18T14:37:31.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:31.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj212 datafile 2024-09-18T14:37:31.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:31.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj213 datafile 2024-09-18T14:37:31.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:31.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj214 datafile 2024-09-18T14:37:31.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:31.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj215 datafile 2024-09-18T14:37:32.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:32.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj216 datafile 2024-09-18T14:37:32.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:32.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj217 datafile 2024-09-18T14:37:32.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:32.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj218 datafile 2024-09-18T14:37:32.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:32.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj219 datafile 2024-09-18T14:37:32.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:32.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj220 datafile 2024-09-18T14:37:32.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:32.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj221 datafile 2024-09-18T14:37:33.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:33.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj222 datafile 2024-09-18T14:37:33.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:33.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj223 datafile 2024-09-18T14:37:33.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:33.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj224 datafile 2024-09-18T14:37:33.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:33.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj225 datafile 2024-09-18T14:37:33.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:33.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj226 datafile 2024-09-18T14:37:33.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:33.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj227 datafile 2024-09-18T14:37:33.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:33.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj228 datafile 2024-09-18T14:37:34.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:34.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj229 datafile 2024-09-18T14:37:34.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:34.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj230 datafile 2024-09-18T14:37:34.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:34.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj231 datafile 2024-09-18T14:37:34.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:34.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj232 datafile 2024-09-18T14:37:34.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:34.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj233 datafile 2024-09-18T14:37:34.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:34.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj234 datafile 2024-09-18T14:37:35.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:35.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj235 datafile 2024-09-18T14:37:35.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:35.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj236 datafile 2024-09-18T14:37:35.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:35.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj237 datafile 2024-09-18T14:37:35.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:35.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj238 datafile 2024-09-18T14:37:35.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:35.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj239 datafile 2024-09-18T14:37:35.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:35.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj240 datafile 2024-09-18T14:37:36.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:36.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj241 datafile 2024-09-18T14:37:36.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:36.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj242 datafile 2024-09-18T14:37:36.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:36.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj243 datafile 2024-09-18T14:37:36.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:36.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj244 datafile 2024-09-18T14:37:36.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:36.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj245 datafile 2024-09-18T14:37:36.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:36.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj246 datafile 2024-09-18T14:37:36.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:36.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj247 datafile 2024-09-18T14:37:37.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:37.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj248 datafile 2024-09-18T14:37:37.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:37.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj249 datafile 2024-09-18T14:37:37.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:37.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj250 datafile 2024-09-18T14:37:37.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:37.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj251 datafile 2024-09-18T14:37:37.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:37.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj252 datafile 2024-09-18T14:37:38.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:38.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj253 datafile 2024-09-18T14:37:38.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:38.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj254 datafile 2024-09-18T14:37:38.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:38.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj255 datafile 2024-09-18T14:37:38.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:38.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj256 datafile 2024-09-18T14:37:38.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:38.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj257 datafile 2024-09-18T14:37:38.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:38.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj258 datafile 2024-09-18T14:37:38.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:38.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj259 datafile 2024-09-18T14:37:39.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:39.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj260 datafile 2024-09-18T14:37:39.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:39.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj261 datafile 2024-09-18T14:37:39.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:39.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj262 datafile 2024-09-18T14:37:39.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:39.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj263 datafile 2024-09-18T14:37:39.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:39.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj264 datafile 2024-09-18T14:37:40.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:40.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj265 datafile 2024-09-18T14:37:40.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:40.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj266 datafile 2024-09-18T14:37:40.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:40.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj267 datafile 2024-09-18T14:37:40.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:40.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj268 datafile 2024-09-18T14:37:40.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:40.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj269 datafile 2024-09-18T14:37:40.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:40.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj270 datafile 2024-09-18T14:37:41.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:41.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj271 datafile 2024-09-18T14:37:41.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:41.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj272 datafile 2024-09-18T14:37:41.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:41.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj273 datafile 2024-09-18T14:37:41.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:41.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj274 datafile 2024-09-18T14:37:41.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:41.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj275 datafile 2024-09-18T14:37:41.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:41.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj276 datafile 2024-09-18T14:37:42.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:42.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj277 datafile 2024-09-18T14:37:42.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:42.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj278 datafile 2024-09-18T14:37:42.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:42.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj279 datafile 2024-09-18T14:37:42.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:42.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj280 datafile 2024-09-18T14:37:42.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:42.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj281 datafile 2024-09-18T14:37:42.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:42.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj282 datafile 2024-09-18T14:37:42.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:42.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj283 datafile 2024-09-18T14:37:43.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:43.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj284 datafile 2024-09-18T14:37:43.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:43.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj285 datafile 2024-09-18T14:37:43.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:43.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj286 datafile 2024-09-18T14:37:43.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:43.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj287 datafile 2024-09-18T14:37:43.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:43.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj288 datafile 2024-09-18T14:37:43.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:43.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj289 datafile 2024-09-18T14:37:43.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:43.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj290 datafile 2024-09-18T14:37:44.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:44.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj291 datafile 2024-09-18T14:37:44.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:44.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj292 datafile 2024-09-18T14:37:44.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:44.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj293 datafile 2024-09-18T14:37:44.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:44.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj294 datafile 2024-09-18T14:37:44.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:44.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj295 datafile 2024-09-18T14:37:44.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:44.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj296 datafile 2024-09-18T14:37:45.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:45.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj297 datafile 2024-09-18T14:37:45.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:45.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj298 datafile 2024-09-18T14:37:45.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:45.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj299 datafile 2024-09-18T14:37:45.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:45.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj300 datafile 2024-09-18T14:37:45.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:45.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj301 datafile 2024-09-18T14:37:45.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:45.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj302 datafile 2024-09-18T14:37:46.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:46.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj303 datafile 2024-09-18T14:37:46.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:46.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj304 datafile 2024-09-18T14:37:46.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:46.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj305 datafile 2024-09-18T14:37:46.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:46.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj306 datafile 2024-09-18T14:37:46.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:46.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj307 datafile 2024-09-18T14:37:46.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:46.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj308 datafile 2024-09-18T14:37:47.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:47.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj309 datafile 2024-09-18T14:37:47.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:47.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj310 datafile 2024-09-18T14:37:47.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:47.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj311 datafile 2024-09-18T14:37:47.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:47.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj312 datafile 2024-09-18T14:37:47.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:47.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj313 datafile 2024-09-18T14:37:47.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:47.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj314 datafile 2024-09-18T14:37:47.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:47.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj315 datafile 2024-09-18T14:37:48.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:48.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj316 datafile 2024-09-18T14:37:48.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:48.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj317 datafile 2024-09-18T14:37:48.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:48.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj318 datafile 2024-09-18T14:37:48.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:48.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj319 datafile 2024-09-18T14:37:48.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:48.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj320 datafile 2024-09-18T14:37:48.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:48.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj321 datafile 2024-09-18T14:37:49.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:49.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj322 datafile 2024-09-18T14:37:49.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:49.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj323 datafile 2024-09-18T14:37:49.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:49.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj324 datafile 2024-09-18T14:37:49.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:49.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj325 datafile 2024-09-18T14:37:49.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:49.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj326 datafile 2024-09-18T14:37:49.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:49.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj327 datafile 2024-09-18T14:37:50.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:50.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj328 datafile 2024-09-18T14:37:50.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:50.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj329 datafile 2024-09-18T14:37:50.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:50.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj330 datafile 2024-09-18T14:37:50.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:50.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj331 datafile 2024-09-18T14:37:50.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:50.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj332 datafile 2024-09-18T14:37:50.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:50.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj333 datafile 2024-09-18T14:37:50.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:50.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj334 datafile 2024-09-18T14:37:51.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:51.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj335 datafile 2024-09-18T14:37:51.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:51.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj336 datafile 2024-09-18T14:37:51.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:51.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj337 datafile 2024-09-18T14:37:51.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:51.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj338 datafile 2024-09-18T14:37:51.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:51.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj339 datafile 2024-09-18T14:37:51.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:51.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj340 datafile 2024-09-18T14:37:52.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:52.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj341 datafile 2024-09-18T14:37:52.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:52.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj342 datafile 2024-09-18T14:37:52.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:52.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj343 datafile 2024-09-18T14:37:52.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:52.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj344 datafile 2024-09-18T14:37:52.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:52.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj345 datafile 2024-09-18T14:37:52.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:52.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj346 datafile 2024-09-18T14:37:52.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:52.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj347 datafile 2024-09-18T14:37:53.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:53.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj348 datafile 2024-09-18T14:37:53.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:53.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj349 datafile 2024-09-18T14:37:53.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:53.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj350 datafile 2024-09-18T14:37:53.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:53.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj351 datafile 2024-09-18T14:37:53.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:53.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj352 datafile 2024-09-18T14:37:53.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:53.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj353 datafile 2024-09-18T14:37:54.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:54.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj354 datafile 2024-09-18T14:37:54.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:54.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj355 datafile 2024-09-18T14:37:54.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:54.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj356 datafile 2024-09-18T14:37:54.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:54.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj357 datafile 2024-09-18T14:37:54.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:54.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj358 datafile 2024-09-18T14:37:54.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:54.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj359 datafile 2024-09-18T14:37:54.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:54.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj360 datafile 2024-09-18T14:37:55.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:55.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj361 datafile 2024-09-18T14:37:55.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:55.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj362 datafile 2024-09-18T14:37:55.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:55.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj363 datafile 2024-09-18T14:37:55.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:55.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj364 datafile 2024-09-18T14:37:55.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:55.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj365 datafile 2024-09-18T14:37:55.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:55.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj366 datafile 2024-09-18T14:37:56.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:56.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj367 datafile 2024-09-18T14:37:56.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:56.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj368 datafile 2024-09-18T14:37:56.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:56.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj369 datafile 2024-09-18T14:37:56.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:56.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj370 datafile 2024-09-18T14:37:56.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:56.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj371 datafile 2024-09-18T14:37:56.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:56.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj372 datafile 2024-09-18T14:37:57.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:57.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj373 datafile 2024-09-18T14:37:57.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:57.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj374 datafile 2024-09-18T14:37:57.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:57.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj375 datafile 2024-09-18T14:37:57.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:57.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj376 datafile 2024-09-18T14:37:57.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:57.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj377 datafile 2024-09-18T14:37:57.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:57.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj378 datafile 2024-09-18T14:37:57.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:57.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj379 datafile 2024-09-18T14:37:58.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:58.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj380 datafile 2024-09-18T14:37:58.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:58.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj381 datafile 2024-09-18T14:37:58.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:58.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj382 datafile 2024-09-18T14:37:58.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:58.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj383 datafile 2024-09-18T14:37:58.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:58.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj384 datafile 2024-09-18T14:37:58.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:58.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj385 datafile 2024-09-18T14:37:58.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:58.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj386 datafile 2024-09-18T14:37:59.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:59.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj387 datafile 2024-09-18T14:37:59.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:59.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj388 datafile 2024-09-18T14:37:59.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:59.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj389 datafile 2024-09-18T14:37:59.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:59.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj390 datafile 2024-09-18T14:37:59.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:59.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj391 datafile 2024-09-18T14:37:59.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:37:59.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj392 datafile 2024-09-18T14:38:00.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:00.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj393 datafile 2024-09-18T14:38:00.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:00.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj394 datafile 2024-09-18T14:38:00.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:00.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj395 datafile 2024-09-18T14:38:00.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:00.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj396 datafile 2024-09-18T14:38:00.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:00.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj397 datafile 2024-09-18T14:38:00.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:00.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj398 datafile 2024-09-18T14:38:01.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:01.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj399 datafile 2024-09-18T14:38:01.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:01.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj400 datafile 2024-09-18T14:38:01.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:01.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj401 datafile 2024-09-18T14:38:01.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:01.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj402 datafile 2024-09-18T14:38:01.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:01.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj403 datafile 2024-09-18T14:38:01.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:01.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj404 datafile 2024-09-18T14:38:01.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:01.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj405 datafile 2024-09-18T14:38:02.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:02.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj406 datafile 2024-09-18T14:38:02.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:02.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj407 datafile 2024-09-18T14:38:02.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:02.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj408 datafile 2024-09-18T14:38:02.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:02.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj409 datafile 2024-09-18T14:38:02.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:02.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj410 datafile 2024-09-18T14:38:02.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:02.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj411 datafile 2024-09-18T14:38:03.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:03.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj412 datafile 2024-09-18T14:38:03.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:03.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj413 datafile 2024-09-18T14:38:03.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:03.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj414 datafile 2024-09-18T14:38:03.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:03.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj415 datafile 2024-09-18T14:38:03.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:03.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj416 datafile 2024-09-18T14:38:03.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:03.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj417 datafile 2024-09-18T14:38:03.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:03.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj418 datafile 2024-09-18T14:38:04.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:04.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj419 datafile 2024-09-18T14:38:04.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:04.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj420 datafile 2024-09-18T14:38:04.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:04.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj421 datafile 2024-09-18T14:38:04.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:04.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj422 datafile 2024-09-18T14:38:04.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:04.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj423 datafile 2024-09-18T14:38:04.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:04.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj424 datafile 2024-09-18T14:38:05.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:05.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj425 datafile 2024-09-18T14:38:05.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:05.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj426 datafile 2024-09-18T14:38:05.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:05.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj427 datafile 2024-09-18T14:38:05.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:05.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj428 datafile 2024-09-18T14:38:05.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:05.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj429 datafile 2024-09-18T14:38:05.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:05.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj430 datafile 2024-09-18T14:38:06.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:06.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj431 datafile 2024-09-18T14:38:06.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:06.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj432 datafile 2024-09-18T14:38:06.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:06.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj433 datafile 2024-09-18T14:38:06.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:06.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj434 datafile 2024-09-18T14:38:06.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:06.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj435 datafile 2024-09-18T14:38:06.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:06.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj436 datafile 2024-09-18T14:38:07.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:07.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj437 datafile 2024-09-18T14:38:07.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:07.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj438 datafile 2024-09-18T14:38:07.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:07.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj439 datafile 2024-09-18T14:38:07.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:07.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj440 datafile 2024-09-18T14:38:07.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:07.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj441 datafile 2024-09-18T14:38:07.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:07.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj442 datafile 2024-09-18T14:38:07.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:07.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj443 datafile 2024-09-18T14:38:08.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:08.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj444 datafile 2024-09-18T14:38:08.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:08.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj445 datafile 2024-09-18T14:38:08.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:08.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj446 datafile 2024-09-18T14:38:08.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:08.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj447 datafile 2024-09-18T14:38:08.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:08.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj448 datafile 2024-09-18T14:38:08.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:08.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj449 datafile 2024-09-18T14:38:09.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:09.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj450 datafile 2024-09-18T14:38:09.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:09.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj451 datafile 2024-09-18T14:38:09.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:09.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj452 datafile 2024-09-18T14:38:09.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:09.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj453 datafile 2024-09-18T14:38:09.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:09.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj454 datafile 2024-09-18T14:38:09.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:09.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj455 datafile 2024-09-18T14:38:10.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:10.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj456 datafile 2024-09-18T14:38:10.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:10.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj457 datafile 2024-09-18T14:38:10.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:10.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj458 datafile 2024-09-18T14:38:10.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:10.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj459 datafile 2024-09-18T14:38:10.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:10.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj460 datafile 2024-09-18T14:38:11.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:11.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj461 datafile 2024-09-18T14:38:11.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:11.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj462 datafile 2024-09-18T14:38:11.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:11.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj463 datafile 2024-09-18T14:38:11.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:11.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj464 datafile 2024-09-18T14:38:11.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:11.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj465 datafile 2024-09-18T14:38:11.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:11.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj466 datafile 2024-09-18T14:38:12.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:12.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj467 datafile 2024-09-18T14:38:12.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:12.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj468 datafile 2024-09-18T14:38:12.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:12.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj469 datafile 2024-09-18T14:38:12.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:12.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj470 datafile 2024-09-18T14:38:12.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:12.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj471 datafile 2024-09-18T14:38:12.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:12.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj472 datafile 2024-09-18T14:38:13.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:13.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj473 datafile 2024-09-18T14:38:13.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:13.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj474 datafile 2024-09-18T14:38:13.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:13.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj475 datafile 2024-09-18T14:38:13.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:13.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj476 datafile 2024-09-18T14:38:13.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:13.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj477 datafile 2024-09-18T14:38:13.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:13.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj478 datafile 2024-09-18T14:38:14.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:14.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj479 datafile 2024-09-18T14:38:14.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:14.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj480 datafile 2024-09-18T14:38:14.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:14.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj481 datafile 2024-09-18T14:38:14.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:14.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj482 datafile 2024-09-18T14:38:14.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:14.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj483 datafile 2024-09-18T14:38:14.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:14.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj484 datafile 2024-09-18T14:38:15.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:15.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj485 datafile 2024-09-18T14:38:15.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:15.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj486 datafile 2024-09-18T14:38:15.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:15.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj487 datafile 2024-09-18T14:38:15.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:15.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj488 datafile 2024-09-18T14:38:15.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:15.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj489 datafile 2024-09-18T14:38:15.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:15.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj490 datafile 2024-09-18T14:38:16.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:16.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj491 datafile 2024-09-18T14:38:16.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:16.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj492 datafile 2024-09-18T14:38:16.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:16.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj493 datafile 2024-09-18T14:38:16.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:16.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj494 datafile 2024-09-18T14:38:16.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:16.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj495 datafile 2024-09-18T14:38:16.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:16.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj496 datafile 2024-09-18T14:38:16.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:16.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj497 datafile 2024-09-18T14:38:17.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:17.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj498 datafile 2024-09-18T14:38:17.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:17.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj499 datafile 2024-09-18T14:38:17.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:17.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj500 datafile 2024-09-18T14:38:17.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:17.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj501 datafile 2024-09-18T14:38:17.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:17.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj502 datafile 2024-09-18T14:38:17.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:17.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj503 datafile 2024-09-18T14:38:17.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:17.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj504 datafile 2024-09-18T14:38:18.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:18.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj505 datafile 2024-09-18T14:38:18.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:18.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj506 datafile 2024-09-18T14:38:18.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:18.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj507 datafile 2024-09-18T14:38:18.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:18.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj508 datafile 2024-09-18T14:38:18.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:18.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj509 datafile 2024-09-18T14:38:18.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:18.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj510 datafile 2024-09-18T14:38:19.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:19.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj511 datafile 2024-09-18T14:38:19.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:19.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj512 datafile 2024-09-18T14:38:19.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:19.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj513 datafile 2024-09-18T14:38:19.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:19.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj514 datafile 2024-09-18T14:38:19.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:19.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj515 datafile 2024-09-18T14:38:19.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:19.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj516 datafile 2024-09-18T14:38:19.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:19.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj517 datafile 2024-09-18T14:38:20.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:20.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj518 datafile 2024-09-18T14:38:20.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:20.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj519 datafile 2024-09-18T14:38:20.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:20.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj520 datafile 2024-09-18T14:38:20.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:20.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj521 datafile 2024-09-18T14:38:20.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:20.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj522 datafile 2024-09-18T14:38:21.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:21.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj523 datafile 2024-09-18T14:38:21.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:21.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj524 datafile 2024-09-18T14:38:21.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:21.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj525 datafile 2024-09-18T14:38:21.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:21.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj526 datafile 2024-09-18T14:38:21.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:21.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj527 datafile 2024-09-18T14:38:21.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:21.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj528 datafile 2024-09-18T14:38:22.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:22.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj529 datafile 2024-09-18T14:38:22.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:22.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj530 datafile 2024-09-18T14:38:22.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:22.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj531 datafile 2024-09-18T14:38:22.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:22.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj532 datafile 2024-09-18T14:38:22.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:22.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj533 datafile 2024-09-18T14:38:22.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:22.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj534 datafile 2024-09-18T14:38:22.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:22.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj535 datafile 2024-09-18T14:38:23.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:23.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj536 datafile 2024-09-18T14:38:23.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:23.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj537 datafile 2024-09-18T14:38:23.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:23.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj538 datafile 2024-09-18T14:38:23.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:23.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj539 datafile 2024-09-18T14:38:23.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:23.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj540 datafile 2024-09-18T14:38:24.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:24.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj541 datafile 2024-09-18T14:38:24.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:24.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj542 datafile 2024-09-18T14:38:24.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:24.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj543 datafile 2024-09-18T14:38:24.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:24.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj544 datafile 2024-09-18T14:38:24.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:24.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj545 datafile 2024-09-18T14:38:24.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:24.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj546 datafile 2024-09-18T14:38:24.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:24.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj547 datafile 2024-09-18T14:38:25.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:25.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj548 datafile 2024-09-18T14:38:25.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:25.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj549 datafile 2024-09-18T14:38:25.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:25.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj550 datafile 2024-09-18T14:38:25.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:25.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj551 datafile 2024-09-18T14:38:25.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:25.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj552 datafile 2024-09-18T14:38:25.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:25.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj553 datafile 2024-09-18T14:38:26.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:26.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj554 datafile 2024-09-18T14:38:26.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:26.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj555 datafile 2024-09-18T14:38:26.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:26.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj556 datafile 2024-09-18T14:38:26.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:26.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj557 datafile 2024-09-18T14:38:26.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:26.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj558 datafile 2024-09-18T14:38:26.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:26.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj559 datafile 2024-09-18T14:38:27.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:27.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj560 datafile 2024-09-18T14:38:27.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:27.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj561 datafile 2024-09-18T14:38:27.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:27.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj562 datafile 2024-09-18T14:38:27.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:27.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj563 datafile 2024-09-18T14:38:27.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:27.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj564 datafile 2024-09-18T14:38:27.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:27.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj565 datafile 2024-09-18T14:38:27.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:27.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj566 datafile 2024-09-18T14:38:28.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:28.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj567 datafile 2024-09-18T14:38:28.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:28.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj568 datafile 2024-09-18T14:38:28.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:28.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj569 datafile 2024-09-18T14:38:28.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:28.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj570 datafile 2024-09-18T14:38:28.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:28.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj571 datafile 2024-09-18T14:38:28.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:28.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj572 datafile 2024-09-18T14:38:29.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:29.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj573 datafile 2024-09-18T14:38:29.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:29.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj574 datafile 2024-09-18T14:38:29.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:29.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj575 datafile 2024-09-18T14:38:29.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:29.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj576 datafile 2024-09-18T14:38:29.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:29.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj577 datafile 2024-09-18T14:38:29.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:29.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj578 datafile 2024-09-18T14:38:30.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:30.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj579 datafile 2024-09-18T14:38:30.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:30.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj580 datafile 2024-09-18T14:38:30.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:30.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj581 datafile 2024-09-18T14:38:30.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:30.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj582 datafile 2024-09-18T14:38:30.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:30.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj583 datafile 2024-09-18T14:38:30.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:30.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj584 datafile 2024-09-18T14:38:31.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:31.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj585 datafile 2024-09-18T14:38:31.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:31.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj586 datafile 2024-09-18T14:38:31.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:31.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj587 datafile 2024-09-18T14:38:31.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:31.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj588 datafile 2024-09-18T14:38:31.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:31.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj589 datafile 2024-09-18T14:38:31.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:31.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj590 datafile 2024-09-18T14:38:31.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:31.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj591 datafile 2024-09-18T14:38:32.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:32.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj592 datafile 2024-09-18T14:38:32.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:32.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj593 datafile 2024-09-18T14:38:32.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:32.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj594 datafile 2024-09-18T14:38:32.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:32.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj595 datafile 2024-09-18T14:38:32.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:32.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj596 datafile 2024-09-18T14:38:32.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:32.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj597 datafile 2024-09-18T14:38:33.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:33.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj598 datafile 2024-09-18T14:38:33.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:33.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj599 datafile 2024-09-18T14:38:33.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:33.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj600 datafile 2024-09-18T14:38:33.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:33.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj601 datafile 2024-09-18T14:38:33.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:33.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj602 datafile 2024-09-18T14:38:33.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:33.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj603 datafile 2024-09-18T14:38:34.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:34.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj604 datafile 2024-09-18T14:38:34.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:34.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj605 datafile 2024-09-18T14:38:34.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:34.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj606 datafile 2024-09-18T14:38:34.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:34.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj607 datafile 2024-09-18T14:38:34.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:34.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj608 datafile 2024-09-18T14:38:34.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:34.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj609 datafile 2024-09-18T14:38:34.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:34.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj610 datafile 2024-09-18T14:38:35.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:35.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj611 datafile 2024-09-18T14:38:35.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:35.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj612 datafile 2024-09-18T14:38:35.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:35.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj613 datafile 2024-09-18T14:38:35.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:35.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj614 datafile 2024-09-18T14:38:35.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:35.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj615 datafile 2024-09-18T14:38:35.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:35.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj616 datafile 2024-09-18T14:38:36.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:36.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj617 datafile 2024-09-18T14:38:36.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:36.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj618 datafile 2024-09-18T14:38:36.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:36.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj619 datafile 2024-09-18T14:38:36.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:36.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj620 datafile 2024-09-18T14:38:36.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:36.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj621 datafile 2024-09-18T14:38:36.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:36.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj622 datafile 2024-09-18T14:38:36.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:36.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj623 datafile 2024-09-18T14:38:37.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:37.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj624 datafile 2024-09-18T14:38:37.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:37.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj625 datafile 2024-09-18T14:38:37.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:37.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj626 datafile 2024-09-18T14:38:37.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:37.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj627 datafile 2024-09-18T14:38:37.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:37.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj628 datafile 2024-09-18T14:38:37.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:37.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj629 datafile 2024-09-18T14:38:37.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:37.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj630 datafile 2024-09-18T14:38:38.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:38.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj631 datafile 2024-09-18T14:38:38.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:38.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj632 datafile 2024-09-18T14:38:38.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:38.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj633 datafile 2024-09-18T14:38:38.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:38.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj634 datafile 2024-09-18T14:38:38.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:38.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj635 datafile 2024-09-18T14:38:38.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:38.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj636 datafile 2024-09-18T14:38:39.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:39.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj637 datafile 2024-09-18T14:38:39.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:39.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj638 datafile 2024-09-18T14:38:39.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:39.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj639 datafile 2024-09-18T14:38:39.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:39.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj640 datafile 2024-09-18T14:38:39.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:39.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj641 datafile 2024-09-18T14:38:39.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:39.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj642 datafile 2024-09-18T14:38:40.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:40.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj643 datafile 2024-09-18T14:38:40.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:40.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj644 datafile 2024-09-18T14:38:40.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:40.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj645 datafile 2024-09-18T14:38:40.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:40.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj646 datafile 2024-09-18T14:38:40.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:40.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj647 datafile 2024-09-18T14:38:40.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:40.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj648 datafile 2024-09-18T14:38:41.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:41.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj649 datafile 2024-09-18T14:38:41.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:41.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj650 datafile 2024-09-18T14:38:41.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:41.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj651 datafile 2024-09-18T14:38:41.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:41.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj652 datafile 2024-09-18T14:38:41.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:41.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj653 datafile 2024-09-18T14:38:41.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:41.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj654 datafile 2024-09-18T14:38:42.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:42.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj655 datafile 2024-09-18T14:38:42.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:42.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj656 datafile 2024-09-18T14:38:42.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:42.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj657 datafile 2024-09-18T14:38:42.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:42.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj658 datafile 2024-09-18T14:38:42.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:42.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj659 datafile 2024-09-18T14:38:43.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:43.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj660 datafile 2024-09-18T14:38:43.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:43.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj661 datafile 2024-09-18T14:38:43.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:43.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj662 datafile 2024-09-18T14:38:43.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:43.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj663 datafile 2024-09-18T14:38:43.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:43.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj664 datafile 2024-09-18T14:38:43.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:43.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj665 datafile 2024-09-18T14:38:43.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:43.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj666 datafile 2024-09-18T14:38:44.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:44.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj667 datafile 2024-09-18T14:38:44.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:44.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj668 datafile 2024-09-18T14:38:44.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:44.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj669 datafile 2024-09-18T14:38:44.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:44.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj670 datafile 2024-09-18T14:38:44.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:44.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj671 datafile 2024-09-18T14:38:44.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:44.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj672 datafile 2024-09-18T14:38:45.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:45.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj673 datafile 2024-09-18T14:38:45.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:45.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj674 datafile 2024-09-18T14:38:45.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:45.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj675 datafile 2024-09-18T14:38:45.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:45.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj676 datafile 2024-09-18T14:38:45.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:45.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj677 datafile 2024-09-18T14:38:45.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:45.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj678 datafile 2024-09-18T14:38:45.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:45.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj679 datafile 2024-09-18T14:38:46.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:46.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj680 datafile 2024-09-18T14:38:46.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:46.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj681 datafile 2024-09-18T14:38:46.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:46.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj682 datafile 2024-09-18T14:38:46.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:46.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj683 datafile 2024-09-18T14:38:46.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:46.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj684 datafile 2024-09-18T14:38:46.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:46.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj685 datafile 2024-09-18T14:38:46.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:46.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj686 datafile 2024-09-18T14:38:47.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:47.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj687 datafile 2024-09-18T14:38:47.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:47.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj688 datafile 2024-09-18T14:38:47.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:47.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj689 datafile 2024-09-18T14:38:47.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:47.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj690 datafile 2024-09-18T14:38:47.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:47.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj691 datafile 2024-09-18T14:38:48.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:48.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj692 datafile 2024-09-18T14:38:48.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:48.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj693 datafile 2024-09-18T14:38:48.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:48.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj694 datafile 2024-09-18T14:38:48.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:48.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj695 datafile 2024-09-18T14:38:48.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:48.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj696 datafile 2024-09-18T14:38:48.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:48.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj697 datafile 2024-09-18T14:38:48.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:48.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj698 datafile 2024-09-18T14:38:48.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:48.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj699 datafile 2024-09-18T14:38:49.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:49.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj700 datafile 2024-09-18T14:38:49.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:49.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj701 datafile 2024-09-18T14:38:49.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:49.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj702 datafile 2024-09-18T14:38:49.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:49.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj703 datafile 2024-09-18T14:38:49.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:49.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj704 datafile 2024-09-18T14:38:49.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:49.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj705 datafile 2024-09-18T14:38:50.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:50.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj706 datafile 2024-09-18T14:38:50.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:50.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj707 datafile 2024-09-18T14:38:50.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:50.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj708 datafile 2024-09-18T14:38:50.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:50.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj709 datafile 2024-09-18T14:38:50.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:50.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj710 datafile 2024-09-18T14:38:50.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:50.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj711 datafile 2024-09-18T14:38:51.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:51.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj712 datafile 2024-09-18T14:38:51.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:51.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj713 datafile 2024-09-18T14:38:51.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:51.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj714 datafile 2024-09-18T14:38:51.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:51.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj715 datafile 2024-09-18T14:38:51.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:51.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj716 datafile 2024-09-18T14:38:51.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:51.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj717 datafile 2024-09-18T14:38:52.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:52.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj718 datafile 2024-09-18T14:38:52.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:52.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj719 datafile 2024-09-18T14:38:52.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:52.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj720 datafile 2024-09-18T14:38:52.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:52.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj721 datafile 2024-09-18T14:38:52.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:52.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj722 datafile 2024-09-18T14:38:52.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:52.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj723 datafile 2024-09-18T14:38:52.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:52.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj724 datafile 2024-09-18T14:38:53.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:53.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj725 datafile 2024-09-18T14:38:53.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:53.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj726 datafile 2024-09-18T14:38:53.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:53.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj727 datafile 2024-09-18T14:38:53.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:53.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj728 datafile 2024-09-18T14:38:53.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:53.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj729 datafile 2024-09-18T14:38:53.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:53.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj730 datafile 2024-09-18T14:38:53.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:53.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj731 datafile 2024-09-18T14:38:54.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:54.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj732 datafile 2024-09-18T14:38:54.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:54.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj733 datafile 2024-09-18T14:38:54.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:54.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj734 datafile 2024-09-18T14:38:54.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:54.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj735 datafile 2024-09-18T14:38:54.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:54.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj736 datafile 2024-09-18T14:38:54.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:54.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj737 datafile 2024-09-18T14:38:55.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:55.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj738 datafile 2024-09-18T14:38:55.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:55.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj739 datafile 2024-09-18T14:38:55.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:55.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj740 datafile 2024-09-18T14:38:55.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:55.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj741 datafile 2024-09-18T14:38:55.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:55.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj742 datafile 2024-09-18T14:38:55.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:55.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj743 datafile 2024-09-18T14:38:55.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:55.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj744 datafile 2024-09-18T14:38:56.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:56.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj745 datafile 2024-09-18T14:38:56.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:56.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj746 datafile 2024-09-18T14:38:56.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:56.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj747 datafile 2024-09-18T14:38:56.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:56.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj748 datafile 2024-09-18T14:38:56.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:56.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj749 datafile 2024-09-18T14:38:56.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:56.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj750 datafile 2024-09-18T14:38:57.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:57.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj751 datafile 2024-09-18T14:38:57.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:57.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj752 datafile 2024-09-18T14:38:57.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:57.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj753 datafile 2024-09-18T14:38:57.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:57.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj754 datafile 2024-09-18T14:38:57.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:57.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj755 datafile 2024-09-18T14:38:57.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:57.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj756 datafile 2024-09-18T14:38:58.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:58.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj757 datafile 2024-09-18T14:38:58.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:58.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj758 datafile 2024-09-18T14:38:58.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:58.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj759 datafile 2024-09-18T14:38:58.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:58.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj760 datafile 2024-09-18T14:38:58.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:58.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj761 datafile 2024-09-18T14:38:58.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:58.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj762 datafile 2024-09-18T14:38:59.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:59.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj763 datafile 2024-09-18T14:38:59.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:59.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj764 datafile 2024-09-18T14:38:59.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:59.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj765 datafile 2024-09-18T14:38:59.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:59.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj766 datafile 2024-09-18T14:38:59.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:59.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj767 datafile 2024-09-18T14:38:59.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:38:59.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj768 datafile 2024-09-18T14:39:00.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:00.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj769 datafile 2024-09-18T14:39:00.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:00.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj770 datafile 2024-09-18T14:39:00.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:00.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj771 datafile 2024-09-18T14:39:00.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:00.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj772 datafile 2024-09-18T14:39:00.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:00.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj773 datafile 2024-09-18T14:39:00.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:00.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj774 datafile 2024-09-18T14:39:00.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:00.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj775 datafile 2024-09-18T14:39:01.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:01.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj776 datafile 2024-09-18T14:39:01.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:01.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj777 datafile 2024-09-18T14:39:01.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:01.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj778 datafile 2024-09-18T14:39:01.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:01.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj779 datafile 2024-09-18T14:39:01.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:01.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj780 datafile 2024-09-18T14:39:01.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:01.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj781 datafile 2024-09-18T14:39:01.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:01.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj782 datafile 2024-09-18T14:39:02.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:02.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj783 datafile 2024-09-18T14:39:02.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:02.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj784 datafile 2024-09-18T14:39:02.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:02.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj785 datafile 2024-09-18T14:39:02.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:02.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj786 datafile 2024-09-18T14:39:02.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:02.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj787 datafile 2024-09-18T14:39:02.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:02.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj788 datafile 2024-09-18T14:39:03.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:03.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj789 datafile 2024-09-18T14:39:03.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:03.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj790 datafile 2024-09-18T14:39:03.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:03.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj791 datafile 2024-09-18T14:39:03.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:03.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj792 datafile 2024-09-18T14:39:03.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:03.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj793 datafile 2024-09-18T14:39:03.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:03.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj794 datafile 2024-09-18T14:39:04.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:04.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj795 datafile 2024-09-18T14:39:04.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:04.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj796 datafile 2024-09-18T14:39:04.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:04.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj797 datafile 2024-09-18T14:39:04.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:04.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj798 datafile 2024-09-18T14:39:04.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:04.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj799 datafile 2024-09-18T14:39:04.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:04.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj800 datafile 2024-09-18T14:39:04.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:04.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj801 datafile 2024-09-18T14:39:05.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:05.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj802 datafile 2024-09-18T14:39:05.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:05.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj803 datafile 2024-09-18T14:39:05.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:05.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj804 datafile 2024-09-18T14:39:05.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:05.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj805 datafile 2024-09-18T14:39:05.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:05.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj806 datafile 2024-09-18T14:39:05.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:05.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj807 datafile 2024-09-18T14:39:06.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:06.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj808 datafile 2024-09-18T14:39:06.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:06.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj809 datafile 2024-09-18T14:39:06.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:06.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj810 datafile 2024-09-18T14:39:06.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:06.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj811 datafile 2024-09-18T14:39:06.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:06.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj812 datafile 2024-09-18T14:39:06.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:06.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj813 datafile 2024-09-18T14:39:07.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:07.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj814 datafile 2024-09-18T14:39:07.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:07.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj815 datafile 2024-09-18T14:39:07.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:07.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj816 datafile 2024-09-18T14:39:07.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:07.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj817 datafile 2024-09-18T14:39:07.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:07.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj818 datafile 2024-09-18T14:39:07.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:07.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj819 datafile 2024-09-18T14:39:08.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:08.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj820 datafile 2024-09-18T14:39:08.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:08.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj821 datafile 2024-09-18T14:39:08.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:08.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj822 datafile 2024-09-18T14:39:08.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:08.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj823 datafile 2024-09-18T14:39:08.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:08.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj824 datafile 2024-09-18T14:39:08.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:08.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj825 datafile 2024-09-18T14:39:08.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:08.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj826 datafile 2024-09-18T14:39:09.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:09.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj827 datafile 2024-09-18T14:39:09.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:09.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj828 datafile 2024-09-18T14:39:09.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:09.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj829 datafile 2024-09-18T14:39:09.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:09.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj830 datafile 2024-09-18T14:39:09.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:09.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj831 datafile 2024-09-18T14:39:09.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:09.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj832 datafile 2024-09-18T14:39:10.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:10.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj833 datafile 2024-09-18T14:39:10.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:10.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj834 datafile 2024-09-18T14:39:10.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:10.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj835 datafile 2024-09-18T14:39:10.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:10.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj836 datafile 2024-09-18T14:39:10.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:10.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj837 datafile 2024-09-18T14:39:10.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:10.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj838 datafile 2024-09-18T14:39:11.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:11.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj839 datafile 2024-09-18T14:39:11.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:11.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj840 datafile 2024-09-18T14:39:11.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:11.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj841 datafile 2024-09-18T14:39:11.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:11.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj842 datafile 2024-09-18T14:39:11.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:11.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj843 datafile 2024-09-18T14:39:11.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:11.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj844 datafile 2024-09-18T14:39:12.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:12.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj845 datafile 2024-09-18T14:39:12.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:12.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj846 datafile 2024-09-18T14:39:12.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:12.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj847 datafile 2024-09-18T14:39:12.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:12.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj848 datafile 2024-09-18T14:39:12.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:12.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj849 datafile 2024-09-18T14:39:12.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:12.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj850 datafile 2024-09-18T14:39:13.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:13.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj851 datafile 2024-09-18T14:39:13.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:13.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj852 datafile 2024-09-18T14:39:13.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:13.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj853 datafile 2024-09-18T14:39:13.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:13.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj854 datafile 2024-09-18T14:39:13.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:13.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj855 datafile 2024-09-18T14:39:13.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:13.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj856 datafile 2024-09-18T14:39:14.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:14.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj857 datafile 2024-09-18T14:39:14.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:14.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj858 datafile 2024-09-18T14:39:14.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:14.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj859 datafile 2024-09-18T14:39:14.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:14.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj860 datafile 2024-09-18T14:39:14.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:14.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj861 datafile 2024-09-18T14:39:14.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:14.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj862 datafile 2024-09-18T14:39:15.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:15.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj863 datafile 2024-09-18T14:39:15.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:15.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj864 datafile 2024-09-18T14:39:15.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:15.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj865 datafile 2024-09-18T14:39:15.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:15.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj866 datafile 2024-09-18T14:39:15.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:15.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj867 datafile 2024-09-18T14:39:15.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:15.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj868 datafile 2024-09-18T14:39:15.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:15.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj869 datafile 2024-09-18T14:39:16.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:16.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj870 datafile 2024-09-18T14:39:16.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:16.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj871 datafile 2024-09-18T14:39:16.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:16.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj872 datafile 2024-09-18T14:39:16.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:16.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj873 datafile 2024-09-18T14:39:16.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:16.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj874 datafile 2024-09-18T14:39:16.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:16.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj875 datafile 2024-09-18T14:39:16.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:16.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj876 datafile 2024-09-18T14:39:17.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:17.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj877 datafile 2024-09-18T14:39:17.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:17.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj878 datafile 2024-09-18T14:39:17.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:17.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj879 datafile 2024-09-18T14:39:17.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:17.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj880 datafile 2024-09-18T14:39:17.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:17.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj881 datafile 2024-09-18T14:39:17.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:17.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj882 datafile 2024-09-18T14:39:18.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:18.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj883 datafile 2024-09-18T14:39:18.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:18.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj884 datafile 2024-09-18T14:39:18.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:18.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj885 datafile 2024-09-18T14:39:18.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:18.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj886 datafile 2024-09-18T14:39:18.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:18.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj887 datafile 2024-09-18T14:39:18.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:18.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj888 datafile 2024-09-18T14:39:19.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:19.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj889 datafile 2024-09-18T14:39:19.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:19.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj890 datafile 2024-09-18T14:39:19.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:19.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj891 datafile 2024-09-18T14:39:19.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:19.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj892 datafile 2024-09-18T14:39:19.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:19.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj893 datafile 2024-09-18T14:39:19.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:19.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj894 datafile 2024-09-18T14:39:19.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:19.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj895 datafile 2024-09-18T14:39:20.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:20.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj896 datafile 2024-09-18T14:39:20.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:20.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj897 datafile 2024-09-18T14:39:20.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:20.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj898 datafile 2024-09-18T14:39:20.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:20.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj899 datafile 2024-09-18T14:39:20.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:20.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj900 datafile 2024-09-18T14:39:20.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:20.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj901 datafile 2024-09-18T14:39:21.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:21.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj902 datafile 2024-09-18T14:39:21.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:21.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj903 datafile 2024-09-18T14:39:21.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:21.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj904 datafile 2024-09-18T14:39:21.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:21.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj905 datafile 2024-09-18T14:39:21.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:21.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj906 datafile 2024-09-18T14:39:21.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:21.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj907 datafile 2024-09-18T14:39:22.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:22.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj908 datafile 2024-09-18T14:39:22.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:22.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj909 datafile 2024-09-18T14:39:22.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:22.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj910 datafile 2024-09-18T14:39:22.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:22.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj911 datafile 2024-09-18T14:39:22.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:22.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj912 datafile 2024-09-18T14:39:22.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:22.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj913 datafile 2024-09-18T14:39:22.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:22.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj914 datafile 2024-09-18T14:39:23.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:23.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj915 datafile 2024-09-18T14:39:23.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:23.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj916 datafile 2024-09-18T14:39:23.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:23.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj917 datafile 2024-09-18T14:39:23.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:23.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj918 datafile 2024-09-18T14:39:23.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:23.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj919 datafile 2024-09-18T14:39:23.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:23.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj920 datafile 2024-09-18T14:39:24.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:24.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj921 datafile 2024-09-18T14:39:24.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:24.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj922 datafile 2024-09-18T14:39:24.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:24.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj923 datafile 2024-09-18T14:39:24.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:24.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj924 datafile 2024-09-18T14:39:24.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:24.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj925 datafile 2024-09-18T14:39:24.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:24.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj926 datafile 2024-09-18T14:39:24.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:24.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj927 datafile 2024-09-18T14:39:25.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:25.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj928 datafile 2024-09-18T14:39:25.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:25.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj929 datafile 2024-09-18T14:39:25.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:25.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj930 datafile 2024-09-18T14:39:25.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:25.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj931 datafile 2024-09-18T14:39:25.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:25.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj932 datafile 2024-09-18T14:39:25.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:25.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj933 datafile 2024-09-18T14:39:26.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:26.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj934 datafile 2024-09-18T14:39:26.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:26.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj935 datafile 2024-09-18T14:39:26.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:26.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj936 datafile 2024-09-18T14:39:26.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:26.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj937 datafile 2024-09-18T14:39:26.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:26.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj938 datafile 2024-09-18T14:39:26.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:26.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj939 datafile 2024-09-18T14:39:26.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:26.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj940 datafile 2024-09-18T14:39:27.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:27.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj941 datafile 2024-09-18T14:39:27.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:27.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj942 datafile 2024-09-18T14:39:27.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:27.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj943 datafile 2024-09-18T14:39:27.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:27.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj944 datafile 2024-09-18T14:39:27.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:27.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj945 datafile 2024-09-18T14:39:27.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:27.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj946 datafile 2024-09-18T14:39:28.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:28.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj947 datafile 2024-09-18T14:39:28.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:28.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj948 datafile 2024-09-18T14:39:28.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:28.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj949 datafile 2024-09-18T14:39:28.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:28.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj950 datafile 2024-09-18T14:39:28.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:28.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj951 datafile 2024-09-18T14:39:28.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:28.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj952 datafile 2024-09-18T14:39:28.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:28.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj953 datafile 2024-09-18T14:39:29.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:29.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj954 datafile 2024-09-18T14:39:29.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:29.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj955 datafile 2024-09-18T14:39:29.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:29.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj956 datafile 2024-09-18T14:39:29.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:29.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj957 datafile 2024-09-18T14:39:29.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:29.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj958 datafile 2024-09-18T14:39:29.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:29.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj959 datafile 2024-09-18T14:39:30.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:30.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj960 datafile 2024-09-18T14:39:30.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:30.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj961 datafile 2024-09-18T14:39:30.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:30.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj962 datafile 2024-09-18T14:39:30.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:30.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj963 datafile 2024-09-18T14:39:30.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:30.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj964 datafile 2024-09-18T14:39:30.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:30.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj965 datafile 2024-09-18T14:39:31.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:31.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj966 datafile 2024-09-18T14:39:31.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:31.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj967 datafile 2024-09-18T14:39:31.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:31.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj968 datafile 2024-09-18T14:39:31.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:31.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj969 datafile 2024-09-18T14:39:31.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:31.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj970 datafile 2024-09-18T14:39:31.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:31.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj971 datafile 2024-09-18T14:39:31.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:31.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj972 datafile 2024-09-18T14:39:32.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:32.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj973 datafile 2024-09-18T14:39:32.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:32.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj974 datafile 2024-09-18T14:39:32.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:32.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj975 datafile 2024-09-18T14:39:32.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:32.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj976 datafile 2024-09-18T14:39:32.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:32.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj977 datafile 2024-09-18T14:39:32.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:32.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj978 datafile 2024-09-18T14:39:33.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:33.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj979 datafile 2024-09-18T14:39:33.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:33.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj980 datafile 2024-09-18T14:39:33.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:33.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj981 datafile 2024-09-18T14:39:33.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:33.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj982 datafile 2024-09-18T14:39:33.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:33.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj983 datafile 2024-09-18T14:39:33.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:33.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj984 datafile 2024-09-18T14:39:33.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:33.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj985 datafile 2024-09-18T14:39:34.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:34.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj986 datafile 2024-09-18T14:39:34.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:34.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj987 datafile 2024-09-18T14:39:34.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:34.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj988 datafile 2024-09-18T14:39:34.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:34.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj989 datafile 2024-09-18T14:39:34.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:34.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj990 datafile 2024-09-18T14:39:34.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:34.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj991 datafile 2024-09-18T14:39:34.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:34.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj992 datafile 2024-09-18T14:39:35.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:35.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj993 datafile 2024-09-18T14:39:35.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:35.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj994 datafile 2024-09-18T14:39:35.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:35.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj995 datafile 2024-09-18T14:39:35.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:35.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj996 datafile 2024-09-18T14:39:35.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:35.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj997 datafile 2024-09-18T14:39:35.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:35.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj998 datafile 2024-09-18T14:39:36.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:36.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj999 datafile 2024-09-18T14:39:36.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:36.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test1 put obj1000 datafile 2024-09-18T14:39:36.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for i in $(seq 1 $POOLS) 2024-09-18T14:39:36.380 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: seq 1 1000 2024-09-18T14:39:36.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:36.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj1 datafile 2024-09-18T14:39:36.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:36.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj2 datafile 2024-09-18T14:39:36.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:36.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj3 datafile 2024-09-18T14:39:36.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:36.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj4 datafile 2024-09-18T14:39:37.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:37.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj5 datafile 2024-09-18T14:39:37.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:37.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj6 datafile 2024-09-18T14:39:37.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:37.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj7 datafile 2024-09-18T14:39:37.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:37.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj8 datafile 2024-09-18T14:39:37.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:37.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj9 datafile 2024-09-18T14:39:37.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:37.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj10 datafile 2024-09-18T14:39:37.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:37.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj11 datafile 2024-09-18T14:39:38.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:38.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj12 datafile 2024-09-18T14:39:38.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:38.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj13 datafile 2024-09-18T14:39:38.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:38.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj14 datafile 2024-09-18T14:39:38.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:38.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj15 datafile 2024-09-18T14:39:38.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:38.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj16 datafile 2024-09-18T14:39:39.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:39.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj17 datafile 2024-09-18T14:39:39.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:39.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj18 datafile 2024-09-18T14:39:39.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:39.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj19 datafile 2024-09-18T14:39:39.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:39.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj20 datafile 2024-09-18T14:39:39.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:39.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj21 datafile 2024-09-18T14:39:39.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:39.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj22 datafile 2024-09-18T14:39:39.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:39.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj23 datafile 2024-09-18T14:39:40.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:40.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj24 datafile 2024-09-18T14:39:40.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:40.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj25 datafile 2024-09-18T14:39:40.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:40.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj26 datafile 2024-09-18T14:39:40.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:40.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj27 datafile 2024-09-18T14:39:40.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:40.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj28 datafile 2024-09-18T14:39:40.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:40.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj29 datafile 2024-09-18T14:39:41.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:41.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj30 datafile 2024-09-18T14:39:41.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:41.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj31 datafile 2024-09-18T14:39:41.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:41.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj32 datafile 2024-09-18T14:39:41.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:41.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj33 datafile 2024-09-18T14:39:41.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:41.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj34 datafile 2024-09-18T14:39:41.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:41.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj35 datafile 2024-09-18T14:39:42.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:42.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj36 datafile 2024-09-18T14:39:42.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:42.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj37 datafile 2024-09-18T14:39:42.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:42.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj38 datafile 2024-09-18T14:39:42.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:42.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj39 datafile 2024-09-18T14:39:42.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:42.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj40 datafile 2024-09-18T14:39:42.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:42.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj41 datafile 2024-09-18T14:39:43.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:43.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj42 datafile 2024-09-18T14:39:43.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:43.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj43 datafile 2024-09-18T14:39:43.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:43.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj44 datafile 2024-09-18T14:39:43.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:43.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj45 datafile 2024-09-18T14:39:43.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:43.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj46 datafile 2024-09-18T14:39:44.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:44.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj47 datafile 2024-09-18T14:39:44.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:44.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj48 datafile 2024-09-18T14:39:44.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:44.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj49 datafile 2024-09-18T14:39:44.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:44.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj50 datafile 2024-09-18T14:39:44.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:44.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj51 datafile 2024-09-18T14:39:44.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:44.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj52 datafile 2024-09-18T14:39:44.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:44.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj53 datafile 2024-09-18T14:39:45.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:45.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj54 datafile 2024-09-18T14:39:45.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:45.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj55 datafile 2024-09-18T14:39:45.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:45.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj56 datafile 2024-09-18T14:39:45.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:45.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj57 datafile 2024-09-18T14:39:45.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:45.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj58 datafile 2024-09-18T14:39:45.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:45.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj59 datafile 2024-09-18T14:39:46.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:46.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj60 datafile 2024-09-18T14:39:46.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:46.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj61 datafile 2024-09-18T14:39:46.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:46.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj62 datafile 2024-09-18T14:39:46.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:46.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj63 datafile 2024-09-18T14:39:46.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:46.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj64 datafile 2024-09-18T14:39:46.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:46.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj65 datafile 2024-09-18T14:39:46.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:46.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj66 datafile 2024-09-18T14:39:47.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:47.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj67 datafile 2024-09-18T14:39:47.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:47.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj68 datafile 2024-09-18T14:39:47.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:47.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj69 datafile 2024-09-18T14:39:47.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:47.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj70 datafile 2024-09-18T14:39:47.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:47.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj71 datafile 2024-09-18T14:39:48.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:48.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj72 datafile 2024-09-18T14:39:48.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:48.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj73 datafile 2024-09-18T14:39:48.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:48.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj74 datafile 2024-09-18T14:39:48.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:48.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj75 datafile 2024-09-18T14:39:48.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:48.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj76 datafile 2024-09-18T14:39:48.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:48.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj77 datafile 2024-09-18T14:39:48.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:48.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj78 datafile 2024-09-18T14:39:49.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:49.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj79 datafile 2024-09-18T14:39:49.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:49.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj80 datafile 2024-09-18T14:39:49.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:49.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj81 datafile 2024-09-18T14:39:49.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:49.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj82 datafile 2024-09-18T14:39:49.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:49.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj83 datafile 2024-09-18T14:39:49.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:49.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj84 datafile 2024-09-18T14:39:50.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:50.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj85 datafile 2024-09-18T14:39:50.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:50.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj86 datafile 2024-09-18T14:39:50.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:50.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj87 datafile 2024-09-18T14:39:50.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:50.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj88 datafile 2024-09-18T14:39:50.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:50.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj89 datafile 2024-09-18T14:39:50.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:50.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj90 datafile 2024-09-18T14:39:50.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:50.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj91 datafile 2024-09-18T14:39:51.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:51.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj92 datafile 2024-09-18T14:39:51.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:51.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj93 datafile 2024-09-18T14:39:51.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:51.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj94 datafile 2024-09-18T14:39:51.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:51.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj95 datafile 2024-09-18T14:39:51.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:51.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj96 datafile 2024-09-18T14:39:51.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:51.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj97 datafile 2024-09-18T14:39:52.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:52.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj98 datafile 2024-09-18T14:39:52.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:52.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj99 datafile 2024-09-18T14:39:52.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:52.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj100 datafile 2024-09-18T14:39:52.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:52.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj101 datafile 2024-09-18T14:39:52.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:52.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj102 datafile 2024-09-18T14:39:52.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:52.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj103 datafile 2024-09-18T14:39:53.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:53.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj104 datafile 2024-09-18T14:39:53.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:53.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj105 datafile 2024-09-18T14:39:53.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:53.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj106 datafile 2024-09-18T14:39:53.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:53.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj107 datafile 2024-09-18T14:39:53.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:53.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj108 datafile 2024-09-18T14:39:53.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:53.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj109 datafile 2024-09-18T14:39:54.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:54.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj110 datafile 2024-09-18T14:39:54.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:54.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj111 datafile 2024-09-18T14:39:54.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:54.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj112 datafile 2024-09-18T14:39:54.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:54.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj113 datafile 2024-09-18T14:39:54.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:54.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj114 datafile 2024-09-18T14:39:54.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:54.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj115 datafile 2024-09-18T14:39:55.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:55.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj116 datafile 2024-09-18T14:39:55.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:55.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj117 datafile 2024-09-18T14:39:55.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:55.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj118 datafile 2024-09-18T14:39:55.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:55.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj119 datafile 2024-09-18T14:39:55.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:55.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj120 datafile 2024-09-18T14:39:55.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:55.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj121 datafile 2024-09-18T14:39:56.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:56.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj122 datafile 2024-09-18T14:39:56.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:56.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj123 datafile 2024-09-18T14:39:56.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:56.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj124 datafile 2024-09-18T14:39:56.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:56.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj125 datafile 2024-09-18T14:39:56.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:56.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj126 datafile 2024-09-18T14:39:56.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:56.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj127 datafile 2024-09-18T14:39:56.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:56.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj128 datafile 2024-09-18T14:39:57.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:57.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj129 datafile 2024-09-18T14:39:57.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:57.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj130 datafile 2024-09-18T14:39:57.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:57.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj131 datafile 2024-09-18T14:39:57.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:57.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj132 datafile 2024-09-18T14:39:57.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:57.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj133 datafile 2024-09-18T14:39:58.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:58.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj134 datafile 2024-09-18T14:39:58.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:58.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj135 datafile 2024-09-18T14:39:58.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:58.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj136 datafile 2024-09-18T14:39:58.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:58.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj137 datafile 2024-09-18T14:39:58.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:58.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj138 datafile 2024-09-18T14:39:58.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:58.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj139 datafile 2024-09-18T14:39:58.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:58.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj140 datafile 2024-09-18T14:39:59.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:59.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj141 datafile 2024-09-18T14:39:59.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:59.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj142 datafile 2024-09-18T14:39:59.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:59.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj143 datafile 2024-09-18T14:39:59.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:59.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj144 datafile 2024-09-18T14:39:59.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:59.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj145 datafile 2024-09-18T14:39:59.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:39:59.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj146 datafile 2024-09-18T14:40:00.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:00.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj147 datafile 2024-09-18T14:40:00.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:00.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj148 datafile 2024-09-18T14:40:00.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:00.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj149 datafile 2024-09-18T14:40:00.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:00.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj150 datafile 2024-09-18T14:40:00.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:00.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj151 datafile 2024-09-18T14:40:00.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:00.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj152 datafile 2024-09-18T14:40:00.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:00.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj153 datafile 2024-09-18T14:40:01.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:01.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj154 datafile 2024-09-18T14:40:01.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:01.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj155 datafile 2024-09-18T14:40:01.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:01.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj156 datafile 2024-09-18T14:40:01.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:01.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj157 datafile 2024-09-18T14:40:01.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:01.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj158 datafile 2024-09-18T14:40:01.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:01.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj159 datafile 2024-09-18T14:40:02.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:02.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj160 datafile 2024-09-18T14:40:02.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:02.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj161 datafile 2024-09-18T14:40:02.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:02.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj162 datafile 2024-09-18T14:40:02.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:02.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj163 datafile 2024-09-18T14:40:02.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:02.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj164 datafile 2024-09-18T14:40:02.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:02.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj165 datafile 2024-09-18T14:40:03.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:03.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj166 datafile 2024-09-18T14:40:03.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:03.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj167 datafile 2024-09-18T14:40:03.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:03.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj168 datafile 2024-09-18T14:40:03.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:03.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj169 datafile 2024-09-18T14:40:03.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:03.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj170 datafile 2024-09-18T14:40:03.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:03.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj171 datafile 2024-09-18T14:40:03.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:03.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj172 datafile 2024-09-18T14:40:04.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:04.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj173 datafile 2024-09-18T14:40:04.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:04.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj174 datafile 2024-09-18T14:40:04.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:04.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj175 datafile 2024-09-18T14:40:04.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:04.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj176 datafile 2024-09-18T14:40:04.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:04.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj177 datafile 2024-09-18T14:40:05.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:05.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj178 datafile 2024-09-18T14:40:05.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:05.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj179 datafile 2024-09-18T14:40:05.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:05.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj180 datafile 2024-09-18T14:40:05.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:05.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj181 datafile 2024-09-18T14:40:05.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:05.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj182 datafile 2024-09-18T14:40:05.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:05.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj183 datafile 2024-09-18T14:40:06.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:06.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj184 datafile 2024-09-18T14:40:06.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:06.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj185 datafile 2024-09-18T14:40:06.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:06.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj186 datafile 2024-09-18T14:40:06.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:06.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj187 datafile 2024-09-18T14:40:06.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:06.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj188 datafile 2024-09-18T14:40:06.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:06.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj189 datafile 2024-09-18T14:40:06.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:06.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj190 datafile 2024-09-18T14:40:07.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:07.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj191 datafile 2024-09-18T14:40:07.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:07.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj192 datafile 2024-09-18T14:40:07.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:07.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj193 datafile 2024-09-18T14:40:07.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:07.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj194 datafile 2024-09-18T14:40:07.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:07.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj195 datafile 2024-09-18T14:40:07.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:07.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj196 datafile 2024-09-18T14:40:07.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:07.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj197 datafile 2024-09-18T14:40:08.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:08.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj198 datafile 2024-09-18T14:40:08.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:08.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj199 datafile 2024-09-18T14:40:08.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:08.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj200 datafile 2024-09-18T14:40:08.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:08.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj201 datafile 2024-09-18T14:40:08.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:08.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj202 datafile 2024-09-18T14:40:09.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:09.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj203 datafile 2024-09-18T14:40:09.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:09.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj204 datafile 2024-09-18T14:40:09.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:09.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj205 datafile 2024-09-18T14:40:09.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:09.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj206 datafile 2024-09-18T14:40:09.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:09.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj207 datafile 2024-09-18T14:40:09.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:09.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj208 datafile 2024-09-18T14:40:09.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:09.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj209 datafile 2024-09-18T14:40:10.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:10.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj210 datafile 2024-09-18T14:40:10.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:10.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj211 datafile 2024-09-18T14:40:10.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:10.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj212 datafile 2024-09-18T14:40:10.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:10.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj213 datafile 2024-09-18T14:40:10.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:10.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj214 datafile 2024-09-18T14:40:11.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:11.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj215 datafile 2024-09-18T14:40:11.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:11.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj216 datafile 2024-09-18T14:40:11.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:11.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj217 datafile 2024-09-18T14:40:11.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:11.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj218 datafile 2024-09-18T14:40:11.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:11.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj219 datafile 2024-09-18T14:40:11.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:11.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj220 datafile 2024-09-18T14:40:12.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:12.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj221 datafile 2024-09-18T14:40:12.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:12.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj222 datafile 2024-09-18T14:40:12.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:12.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj223 datafile 2024-09-18T14:40:12.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:12.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj224 datafile 2024-09-18T14:40:12.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:12.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj225 datafile 2024-09-18T14:40:12.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:12.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj226 datafile 2024-09-18T14:40:12.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:12.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj227 datafile 2024-09-18T14:40:13.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:13.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj228 datafile 2024-09-18T14:40:13.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:13.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj229 datafile 2024-09-18T14:40:13.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:13.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj230 datafile 2024-09-18T14:40:13.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:13.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj231 datafile 2024-09-18T14:40:13.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:13.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj232 datafile 2024-09-18T14:40:14.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:14.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj233 datafile 2024-09-18T14:40:14.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:14.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj234 datafile 2024-09-18T14:40:14.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:14.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj235 datafile 2024-09-18T14:40:14.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:14.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj236 datafile 2024-09-18T14:40:14.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:14.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj237 datafile 2024-09-18T14:40:14.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:14.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj238 datafile 2024-09-18T14:40:15.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:15.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj239 datafile 2024-09-18T14:40:15.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:15.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj240 datafile 2024-09-18T14:40:15.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:15.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj241 datafile 2024-09-18T14:40:15.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:15.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj242 datafile 2024-09-18T14:40:15.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:15.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj243 datafile 2024-09-18T14:40:15.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:15.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj244 datafile 2024-09-18T14:40:16.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:16.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj245 datafile 2024-09-18T14:40:16.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:16.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj246 datafile 2024-09-18T14:40:16.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:16.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj247 datafile 2024-09-18T14:40:16.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:16.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj248 datafile 2024-09-18T14:40:16.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:16.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj249 datafile 2024-09-18T14:40:16.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:16.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj250 datafile 2024-09-18T14:40:16.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:16.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj251 datafile 2024-09-18T14:40:17.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:17.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj252 datafile 2024-09-18T14:40:17.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:17.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj253 datafile 2024-09-18T14:40:17.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:17.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj254 datafile 2024-09-18T14:40:17.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:17.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj255 datafile 2024-09-18T14:40:17.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:17.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj256 datafile 2024-09-18T14:40:18.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:18.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj257 datafile 2024-09-18T14:40:18.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:18.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj258 datafile 2024-09-18T14:40:18.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:18.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj259 datafile 2024-09-18T14:40:18.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:18.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj260 datafile 2024-09-18T14:40:18.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:18.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj261 datafile 2024-09-18T14:40:18.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:18.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj262 datafile 2024-09-18T14:40:19.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:19.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj263 datafile 2024-09-18T14:40:19.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:19.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj264 datafile 2024-09-18T14:40:19.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:19.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj265 datafile 2024-09-18T14:40:19.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:19.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj266 datafile 2024-09-18T14:40:19.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:19.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj267 datafile 2024-09-18T14:40:19.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:19.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj268 datafile 2024-09-18T14:40:20.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:20.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj269 datafile 2024-09-18T14:40:20.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:20.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj270 datafile 2024-09-18T14:40:20.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:20.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj271 datafile 2024-09-18T14:40:20.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:20.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj272 datafile 2024-09-18T14:40:20.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:20.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj273 datafile 2024-09-18T14:40:21.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:21.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj274 datafile 2024-09-18T14:40:21.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:21.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj275 datafile 2024-09-18T14:40:21.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:21.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj276 datafile 2024-09-18T14:40:21.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:21.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj277 datafile 2024-09-18T14:40:21.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:21.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj278 datafile 2024-09-18T14:40:21.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:21.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj279 datafile 2024-09-18T14:40:22.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:22.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj280 datafile 2024-09-18T14:40:22.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:22.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj281 datafile 2024-09-18T14:40:22.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:22.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj282 datafile 2024-09-18T14:40:22.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:22.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj283 datafile 2024-09-18T14:40:22.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:22.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj284 datafile 2024-09-18T14:40:22.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:22.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj285 datafile 2024-09-18T14:40:23.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:23.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj286 datafile 2024-09-18T14:40:23.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:23.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj287 datafile 2024-09-18T14:40:23.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:23.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj288 datafile 2024-09-18T14:40:23.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:23.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj289 datafile 2024-09-18T14:40:23.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:23.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj290 datafile 2024-09-18T14:40:23.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:23.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj291 datafile 2024-09-18T14:40:24.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:24.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj292 datafile 2024-09-18T14:40:24.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:24.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj293 datafile 2024-09-18T14:40:24.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:24.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj294 datafile 2024-09-18T14:40:24.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:24.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj295 datafile 2024-09-18T14:40:24.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:24.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj296 datafile 2024-09-18T14:40:25.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:25.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj297 datafile 2024-09-18T14:40:25.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:25.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj298 datafile 2024-09-18T14:40:25.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:25.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj299 datafile 2024-09-18T14:40:25.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:25.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj300 datafile 2024-09-18T14:40:25.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:25.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj301 datafile 2024-09-18T14:40:25.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:25.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj302 datafile 2024-09-18T14:40:26.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:26.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj303 datafile 2024-09-18T14:40:26.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:26.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj304 datafile 2024-09-18T14:40:26.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:26.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj305 datafile 2024-09-18T14:40:26.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:26.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj306 datafile 2024-09-18T14:40:26.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:26.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj307 datafile 2024-09-18T14:40:26.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:26.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj308 datafile 2024-09-18T14:40:26.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:26.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj309 datafile 2024-09-18T14:40:27.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:27.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj310 datafile 2024-09-18T14:40:27.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:27.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj311 datafile 2024-09-18T14:40:27.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:27.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj312 datafile 2024-09-18T14:40:27.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:27.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj313 datafile 2024-09-18T14:40:27.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:27.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj314 datafile 2024-09-18T14:40:27.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:27.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj315 datafile 2024-09-18T14:40:28.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:28.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj316 datafile 2024-09-18T14:40:28.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:28.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj317 datafile 2024-09-18T14:40:28.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:28.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj318 datafile 2024-09-18T14:40:28.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:28.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj319 datafile 2024-09-18T14:40:28.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:28.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj320 datafile 2024-09-18T14:40:28.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:28.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj321 datafile 2024-09-18T14:40:28.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:28.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj322 datafile 2024-09-18T14:40:29.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:29.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj323 datafile 2024-09-18T14:40:29.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:29.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj324 datafile 2024-09-18T14:40:29.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:29.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj325 datafile 2024-09-18T14:40:29.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:29.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj326 datafile 2024-09-18T14:40:29.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:29.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj327 datafile 2024-09-18T14:40:29.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:29.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj328 datafile 2024-09-18T14:40:30.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:30.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj329 datafile 2024-09-18T14:40:30.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:30.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj330 datafile 2024-09-18T14:40:30.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:30.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj331 datafile 2024-09-18T14:40:30.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:30.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj332 datafile 2024-09-18T14:40:30.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:30.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj333 datafile 2024-09-18T14:40:30.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:30.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj334 datafile 2024-09-18T14:40:31.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:31.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj335 datafile 2024-09-18T14:40:31.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:31.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj336 datafile 2024-09-18T14:40:31.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:31.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj337 datafile 2024-09-18T14:40:31.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:31.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj338 datafile 2024-09-18T14:40:31.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:31.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj339 datafile 2024-09-18T14:40:31.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:31.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj340 datafile 2024-09-18T14:40:31.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:31.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj341 datafile 2024-09-18T14:40:32.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:32.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj342 datafile 2024-09-18T14:40:32.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:32.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj343 datafile 2024-09-18T14:40:32.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:32.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj344 datafile 2024-09-18T14:40:32.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:32.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj345 datafile 2024-09-18T14:40:32.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:32.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj346 datafile 2024-09-18T14:40:32.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:32.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj347 datafile 2024-09-18T14:40:33.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:33.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj348 datafile 2024-09-18T14:40:33.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:33.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj349 datafile 2024-09-18T14:40:33.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:33.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj350 datafile 2024-09-18T14:40:33.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:33.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj351 datafile 2024-09-18T14:40:33.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:33.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj352 datafile 2024-09-18T14:40:33.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:33.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj353 datafile 2024-09-18T14:40:34.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:34.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj354 datafile 2024-09-18T14:40:34.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:34.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj355 datafile 2024-09-18T14:40:34.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:34.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj356 datafile 2024-09-18T14:40:34.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:34.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj357 datafile 2024-09-18T14:40:34.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:34.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj358 datafile 2024-09-18T14:40:34.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:34.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj359 datafile 2024-09-18T14:40:35.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:35.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj360 datafile 2024-09-18T14:40:35.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:35.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj361 datafile 2024-09-18T14:40:35.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:35.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj362 datafile 2024-09-18T14:40:35.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:35.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj363 datafile 2024-09-18T14:40:35.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:35.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj364 datafile 2024-09-18T14:40:35.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:35.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj365 datafile 2024-09-18T14:40:36.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:36.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj366 datafile 2024-09-18T14:40:36.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:36.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj367 datafile 2024-09-18T14:40:36.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:36.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj368 datafile 2024-09-18T14:40:36.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:36.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj369 datafile 2024-09-18T14:40:36.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:36.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj370 datafile 2024-09-18T14:40:36.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:36.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj371 datafile 2024-09-18T14:40:37.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:37.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj372 datafile 2024-09-18T14:40:37.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:37.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj373 datafile 2024-09-18T14:40:37.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:37.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj374 datafile 2024-09-18T14:40:37.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:37.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj375 datafile 2024-09-18T14:40:37.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:37.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj376 datafile 2024-09-18T14:40:37.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:37.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj377 datafile 2024-09-18T14:40:37.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:37.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj378 datafile 2024-09-18T14:40:38.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:38.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj379 datafile 2024-09-18T14:40:38.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:38.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj380 datafile 2024-09-18T14:40:38.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:38.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj381 datafile 2024-09-18T14:40:38.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:38.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj382 datafile 2024-09-18T14:40:38.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:38.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj383 datafile 2024-09-18T14:40:38.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:38.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj384 datafile 2024-09-18T14:40:39.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:39.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj385 datafile 2024-09-18T14:40:39.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:39.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj386 datafile 2024-09-18T14:40:39.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:39.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj387 datafile 2024-09-18T14:40:39.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:39.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj388 datafile 2024-09-18T14:40:39.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:39.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj389 datafile 2024-09-18T14:40:39.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:39.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj390 datafile 2024-09-18T14:40:40.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:40.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj391 datafile 2024-09-18T14:40:40.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:40.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj392 datafile 2024-09-18T14:40:40.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:40.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj393 datafile 2024-09-18T14:40:40.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:40.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj394 datafile 2024-09-18T14:40:40.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:40.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj395 datafile 2024-09-18T14:40:40.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:40.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj396 datafile 2024-09-18T14:40:40.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:40.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj397 datafile 2024-09-18T14:40:41.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:41.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj398 datafile 2024-09-18T14:40:41.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:41.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj399 datafile 2024-09-18T14:40:41.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:41.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj400 datafile 2024-09-18T14:40:41.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:41.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj401 datafile 2024-09-18T14:40:41.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:41.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj402 datafile 2024-09-18T14:40:41.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:41.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj403 datafile 2024-09-18T14:40:42.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:42.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj404 datafile 2024-09-18T14:40:42.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:42.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj405 datafile 2024-09-18T14:40:42.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:42.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj406 datafile 2024-09-18T14:40:42.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:42.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj407 datafile 2024-09-18T14:40:42.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:42.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj408 datafile 2024-09-18T14:40:42.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:42.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj409 datafile 2024-09-18T14:40:43.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:43.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj410 datafile 2024-09-18T14:40:43.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:43.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj411 datafile 2024-09-18T14:40:43.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:43.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj412 datafile 2024-09-18T14:40:43.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:43.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj413 datafile 2024-09-18T14:40:43.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:43.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj414 datafile 2024-09-18T14:40:43.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:43.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj415 datafile 2024-09-18T14:40:43.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:43.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj416 datafile 2024-09-18T14:40:44.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:44.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj417 datafile 2024-09-18T14:40:44.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:44.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj418 datafile 2024-09-18T14:40:44.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:44.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj419 datafile 2024-09-18T14:40:44.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:44.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj420 datafile 2024-09-18T14:40:45.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:45.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj421 datafile 2024-09-18T14:40:45.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:45.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj422 datafile 2024-09-18T14:40:45.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:45.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj423 datafile 2024-09-18T14:40:45.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:45.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj424 datafile 2024-09-18T14:40:45.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:45.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj425 datafile 2024-09-18T14:40:46.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:46.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj426 datafile 2024-09-18T14:40:46.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:46.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj427 datafile 2024-09-18T14:40:46.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:46.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj428 datafile 2024-09-18T14:40:46.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:46.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj429 datafile 2024-09-18T14:40:46.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:46.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj430 datafile 2024-09-18T14:40:46.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:46.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj431 datafile 2024-09-18T14:40:46.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:46.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj432 datafile 2024-09-18T14:40:47.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:47.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj433 datafile 2024-09-18T14:40:47.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:47.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj434 datafile 2024-09-18T14:40:47.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:47.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj435 datafile 2024-09-18T14:40:47.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:47.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj436 datafile 2024-09-18T14:40:47.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:47.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj437 datafile 2024-09-18T14:40:47.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:47.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj438 datafile 2024-09-18T14:40:48.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:48.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj439 datafile 2024-09-18T14:40:48.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:48.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj440 datafile 2024-09-18T14:40:48.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:48.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj441 datafile 2024-09-18T14:40:48.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:48.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj442 datafile 2024-09-18T14:40:48.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:48.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj443 datafile 2024-09-18T14:40:48.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:48.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj444 datafile 2024-09-18T14:40:49.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:49.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj445 datafile 2024-09-18T14:40:49.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:49.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj446 datafile 2024-09-18T14:40:49.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:49.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj447 datafile 2024-09-18T14:40:49.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:49.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj448 datafile 2024-09-18T14:40:49.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:49.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj449 datafile 2024-09-18T14:40:49.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:49.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj450 datafile 2024-09-18T14:40:50.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:50.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj451 datafile 2024-09-18T14:40:50.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:50.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj452 datafile 2024-09-18T14:40:50.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:50.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj453 datafile 2024-09-18T14:40:50.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:50.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj454 datafile 2024-09-18T14:40:50.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:50.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj455 datafile 2024-09-18T14:40:50.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:50.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj456 datafile 2024-09-18T14:40:51.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:51.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj457 datafile 2024-09-18T14:40:51.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:51.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj458 datafile 2024-09-18T14:40:51.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:51.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj459 datafile 2024-09-18T14:40:51.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:51.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj460 datafile 2024-09-18T14:40:51.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:51.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj461 datafile 2024-09-18T14:40:51.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:51.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj462 datafile 2024-09-18T14:40:51.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:51.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj463 datafile 2024-09-18T14:40:52.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:52.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj464 datafile 2024-09-18T14:40:52.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:52.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj465 datafile 2024-09-18T14:40:52.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:52.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj466 datafile 2024-09-18T14:40:52.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:52.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj467 datafile 2024-09-18T14:40:52.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:52.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj468 datafile 2024-09-18T14:40:52.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:52.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj469 datafile 2024-09-18T14:40:53.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:53.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj470 datafile 2024-09-18T14:40:53.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:53.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj471 datafile 2024-09-18T14:40:53.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:53.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj472 datafile 2024-09-18T14:40:53.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:53.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj473 datafile 2024-09-18T14:40:53.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:53.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj474 datafile 2024-09-18T14:40:53.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:53.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj475 datafile 2024-09-18T14:40:54.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:54.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj476 datafile 2024-09-18T14:40:54.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:54.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj477 datafile 2024-09-18T14:40:54.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:54.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj478 datafile 2024-09-18T14:40:54.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:54.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj479 datafile 2024-09-18T14:40:54.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:54.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj480 datafile 2024-09-18T14:40:54.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:54.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj481 datafile 2024-09-18T14:40:55.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:55.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj482 datafile 2024-09-18T14:40:55.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:55.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj483 datafile 2024-09-18T14:40:55.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:55.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj484 datafile 2024-09-18T14:40:55.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:55.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj485 datafile 2024-09-18T14:40:55.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:55.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj486 datafile 2024-09-18T14:40:56.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:56.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj487 datafile 2024-09-18T14:40:56.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:56.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj488 datafile 2024-09-18T14:40:56.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:56.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj489 datafile 2024-09-18T14:40:56.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:56.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj490 datafile 2024-09-18T14:40:56.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:56.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj491 datafile 2024-09-18T14:40:56.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:56.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj492 datafile 2024-09-18T14:40:57.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:57.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj493 datafile 2024-09-18T14:40:57.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:57.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj494 datafile 2024-09-18T14:40:57.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:57.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj495 datafile 2024-09-18T14:40:57.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:57.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj496 datafile 2024-09-18T14:40:57.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:57.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj497 datafile 2024-09-18T14:40:57.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:57.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj498 datafile 2024-09-18T14:40:58.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:58.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj499 datafile 2024-09-18T14:40:58.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:58.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj500 datafile 2024-09-18T14:40:58.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:58.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj501 datafile 2024-09-18T14:40:58.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:58.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj502 datafile 2024-09-18T14:40:58.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:58.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj503 datafile 2024-09-18T14:40:58.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:58.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj504 datafile 2024-09-18T14:40:59.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:59.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj505 datafile 2024-09-18T14:40:59.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:59.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj506 datafile 2024-09-18T14:40:59.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:59.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj507 datafile 2024-09-18T14:40:59.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:59.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj508 datafile 2024-09-18T14:40:59.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:59.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj509 datafile 2024-09-18T14:40:59.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:40:59.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj510 datafile 2024-09-18T14:41:00.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:00.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj511 datafile 2024-09-18T14:41:00.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:00.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj512 datafile 2024-09-18T14:41:00.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:00.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj513 datafile 2024-09-18T14:41:00.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:00.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj514 datafile 2024-09-18T14:41:00.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:00.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj515 datafile 2024-09-18T14:41:00.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:00.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj516 datafile 2024-09-18T14:41:00.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:00.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj517 datafile 2024-09-18T14:41:01.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:01.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj518 datafile 2024-09-18T14:41:01.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:01.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj519 datafile 2024-09-18T14:41:01.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:01.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj520 datafile 2024-09-18T14:41:01.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:01.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj521 datafile 2024-09-18T14:41:01.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:01.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj522 datafile 2024-09-18T14:41:01.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:01.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj523 datafile 2024-09-18T14:41:02.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:02.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj524 datafile 2024-09-18T14:41:02.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:02.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj525 datafile 2024-09-18T14:41:02.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:02.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj526 datafile 2024-09-18T14:41:02.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:02.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj527 datafile 2024-09-18T14:41:02.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:02.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj528 datafile 2024-09-18T14:41:02.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:02.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj529 datafile 2024-09-18T14:41:02.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:02.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj530 datafile 2024-09-18T14:41:03.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:03.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj531 datafile 2024-09-18T14:41:03.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:03.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj532 datafile 2024-09-18T14:41:03.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:03.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj533 datafile 2024-09-18T14:41:03.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:03.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj534 datafile 2024-09-18T14:41:03.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:03.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj535 datafile 2024-09-18T14:41:03.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:03.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj536 datafile 2024-09-18T14:41:03.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:03.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj537 datafile 2024-09-18T14:41:04.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:04.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj538 datafile 2024-09-18T14:41:04.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:04.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj539 datafile 2024-09-18T14:41:04.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:04.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj540 datafile 2024-09-18T14:41:04.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:04.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj541 datafile 2024-09-18T14:41:04.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:04.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj542 datafile 2024-09-18T14:41:04.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:04.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj543 datafile 2024-09-18T14:41:05.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:05.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj544 datafile 2024-09-18T14:41:05.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:05.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj545 datafile 2024-09-18T14:41:05.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:05.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj546 datafile 2024-09-18T14:41:05.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:05.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj547 datafile 2024-09-18T14:41:05.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:05.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj548 datafile 2024-09-18T14:41:05.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:05.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj549 datafile 2024-09-18T14:41:05.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:05.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj550 datafile 2024-09-18T14:41:06.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:06.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj551 datafile 2024-09-18T14:41:06.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:06.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj552 datafile 2024-09-18T14:41:06.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:06.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj553 datafile 2024-09-18T14:41:06.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:06.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj554 datafile 2024-09-18T14:41:06.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:06.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj555 datafile 2024-09-18T14:41:06.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:06.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj556 datafile 2024-09-18T14:41:07.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:07.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj557 datafile 2024-09-18T14:41:07.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:07.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj558 datafile 2024-09-18T14:41:07.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:07.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj559 datafile 2024-09-18T14:41:07.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:07.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj560 datafile 2024-09-18T14:41:07.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:07.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj561 datafile 2024-09-18T14:41:07.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:07.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj562 datafile 2024-09-18T14:41:07.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:07.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj563 datafile 2024-09-18T14:41:08.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:08.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj564 datafile 2024-09-18T14:41:08.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:08.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj565 datafile 2024-09-18T14:41:08.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:08.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj566 datafile 2024-09-18T14:41:08.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:08.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj567 datafile 2024-09-18T14:41:08.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:08.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj568 datafile 2024-09-18T14:41:08.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:08.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj569 datafile 2024-09-18T14:41:09.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:09.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj570 datafile 2024-09-18T14:41:09.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:09.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj571 datafile 2024-09-18T14:41:09.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:09.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj572 datafile 2024-09-18T14:41:09.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:09.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj573 datafile 2024-09-18T14:41:09.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:09.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj574 datafile 2024-09-18T14:41:09.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:09.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj575 datafile 2024-09-18T14:41:09.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:09.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj576 datafile 2024-09-18T14:41:10.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:10.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj577 datafile 2024-09-18T14:41:10.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:10.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj578 datafile 2024-09-18T14:41:10.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:10.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj579 datafile 2024-09-18T14:41:10.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:10.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj580 datafile 2024-09-18T14:41:10.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:10.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj581 datafile 2024-09-18T14:41:10.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:10.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj582 datafile 2024-09-18T14:41:11.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:11.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj583 datafile 2024-09-18T14:41:11.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:11.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj584 datafile 2024-09-18T14:41:11.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:11.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj585 datafile 2024-09-18T14:41:11.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:11.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj586 datafile 2024-09-18T14:41:11.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:11.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj587 datafile 2024-09-18T14:41:11.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:11.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj588 datafile 2024-09-18T14:41:11.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:11.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj589 datafile 2024-09-18T14:41:12.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:12.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj590 datafile 2024-09-18T14:41:12.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:12.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj591 datafile 2024-09-18T14:41:12.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:12.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj592 datafile 2024-09-18T14:41:12.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:12.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj593 datafile 2024-09-18T14:41:12.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:12.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj594 datafile 2024-09-18T14:41:12.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:12.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj595 datafile 2024-09-18T14:41:13.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:13.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj596 datafile 2024-09-18T14:41:13.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:13.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj597 datafile 2024-09-18T14:41:13.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:13.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj598 datafile 2024-09-18T14:41:13.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:13.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj599 datafile 2024-09-18T14:41:13.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:13.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj600 datafile 2024-09-18T14:41:13.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:13.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj601 datafile 2024-09-18T14:41:13.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:13.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj602 datafile 2024-09-18T14:41:14.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:14.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj603 datafile 2024-09-18T14:41:14.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:14.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj604 datafile 2024-09-18T14:41:14.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:14.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj605 datafile 2024-09-18T14:41:14.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:14.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj606 datafile 2024-09-18T14:41:14.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:14.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj607 datafile 2024-09-18T14:41:14.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:14.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj608 datafile 2024-09-18T14:41:14.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:14.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj609 datafile 2024-09-18T14:41:15.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:15.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj610 datafile 2024-09-18T14:41:15.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:15.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj611 datafile 2024-09-18T14:41:15.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:15.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj612 datafile 2024-09-18T14:41:15.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:15.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj613 datafile 2024-09-18T14:41:15.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:15.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj614 datafile 2024-09-18T14:41:16.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:16.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj615 datafile 2024-09-18T14:41:16.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:16.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj616 datafile 2024-09-18T14:41:16.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:16.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj617 datafile 2024-09-18T14:41:16.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:16.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj618 datafile 2024-09-18T14:41:16.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:16.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj619 datafile 2024-09-18T14:41:16.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:16.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj620 datafile 2024-09-18T14:41:17.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:17.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj621 datafile 2024-09-18T14:41:17.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:17.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj622 datafile 2024-09-18T14:41:17.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:17.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj623 datafile 2024-09-18T14:41:17.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:17.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj624 datafile 2024-09-18T14:41:17.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:17.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj625 datafile 2024-09-18T14:41:17.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:17.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj626 datafile 2024-09-18T14:41:18.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:18.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj627 datafile 2024-09-18T14:41:18.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:18.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj628 datafile 2024-09-18T14:41:18.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:18.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj629 datafile 2024-09-18T14:41:18.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:18.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj630 datafile 2024-09-18T14:41:18.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:18.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj631 datafile 2024-09-18T14:41:18.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:18.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj632 datafile 2024-09-18T14:41:18.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:18.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj633 datafile 2024-09-18T14:41:19.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:19.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj634 datafile 2024-09-18T14:41:19.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:19.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj635 datafile 2024-09-18T14:41:19.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:19.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj636 datafile 2024-09-18T14:41:19.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:19.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj637 datafile 2024-09-18T14:41:19.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:19.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj638 datafile 2024-09-18T14:41:19.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:19.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj639 datafile 2024-09-18T14:41:20.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:20.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj640 datafile 2024-09-18T14:41:20.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:20.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj641 datafile 2024-09-18T14:41:20.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:20.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj642 datafile 2024-09-18T14:41:20.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:20.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj643 datafile 2024-09-18T14:41:20.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:20.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj644 datafile 2024-09-18T14:41:20.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:20.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj645 datafile 2024-09-18T14:41:21.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:21.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj646 datafile 2024-09-18T14:41:21.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:21.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj647 datafile 2024-09-18T14:41:21.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:21.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj648 datafile 2024-09-18T14:41:21.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:21.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj649 datafile 2024-09-18T14:41:21.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:21.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj650 datafile 2024-09-18T14:41:21.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:21.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj651 datafile 2024-09-18T14:41:22.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:22.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj652 datafile 2024-09-18T14:41:22.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:22.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj653 datafile 2024-09-18T14:41:22.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:22.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj654 datafile 2024-09-18T14:41:22.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:22.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj655 datafile 2024-09-18T14:41:22.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:22.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj656 datafile 2024-09-18T14:41:22.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:22.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj657 datafile 2024-09-18T14:41:23.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:23.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj658 datafile 2024-09-18T14:41:23.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:23.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj659 datafile 2024-09-18T14:41:23.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:23.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj660 datafile 2024-09-18T14:41:23.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:23.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj661 datafile 2024-09-18T14:41:23.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:23.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj662 datafile 2024-09-18T14:41:23.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:23.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj663 datafile 2024-09-18T14:41:23.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:23.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj664 datafile 2024-09-18T14:41:24.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:24.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj665 datafile 2024-09-18T14:41:24.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:24.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj666 datafile 2024-09-18T14:41:24.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:24.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj667 datafile 2024-09-18T14:41:24.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:24.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj668 datafile 2024-09-18T14:41:24.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:24.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj669 datafile 2024-09-18T14:41:24.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:24.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj670 datafile 2024-09-18T14:41:25.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:25.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj671 datafile 2024-09-18T14:41:25.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:25.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj672 datafile 2024-09-18T14:41:25.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:25.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj673 datafile 2024-09-18T14:41:25.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:25.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj674 datafile 2024-09-18T14:41:25.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:25.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj675 datafile 2024-09-18T14:41:25.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:25.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj676 datafile 2024-09-18T14:41:25.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:25.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj677 datafile 2024-09-18T14:41:26.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:26.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj678 datafile 2024-09-18T14:41:26.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:26.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj679 datafile 2024-09-18T14:41:26.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:26.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj680 datafile 2024-09-18T14:41:26.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:26.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj681 datafile 2024-09-18T14:41:26.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:26.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj682 datafile 2024-09-18T14:41:27.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:27.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj683 datafile 2024-09-18T14:41:27.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:27.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj684 datafile 2024-09-18T14:41:27.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:27.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj685 datafile 2024-09-18T14:41:27.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:27.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj686 datafile 2024-09-18T14:41:27.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:27.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj687 datafile 2024-09-18T14:41:27.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:27.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj688 datafile 2024-09-18T14:41:27.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:27.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj689 datafile 2024-09-18T14:41:28.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:28.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj690 datafile 2024-09-18T14:41:28.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:28.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj691 datafile 2024-09-18T14:41:28.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:28.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj692 datafile 2024-09-18T14:41:28.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:28.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj693 datafile 2024-09-18T14:41:28.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:28.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj694 datafile 2024-09-18T14:41:28.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:28.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj695 datafile 2024-09-18T14:41:28.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:28.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj696 datafile 2024-09-18T14:41:29.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:29.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj697 datafile 2024-09-18T14:41:29.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:29.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj698 datafile 2024-09-18T14:41:29.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:29.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj699 datafile 2024-09-18T14:41:29.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:29.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj700 datafile 2024-09-18T14:41:29.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:29.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj701 datafile 2024-09-18T14:41:29.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:29.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj702 datafile 2024-09-18T14:41:30.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:30.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj703 datafile 2024-09-18T14:41:30.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:30.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj704 datafile 2024-09-18T14:41:30.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:30.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj705 datafile 2024-09-18T14:41:30.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:30.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj706 datafile 2024-09-18T14:41:30.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:30.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj707 datafile 2024-09-18T14:41:30.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:30.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj708 datafile 2024-09-18T14:41:31.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:31.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj709 datafile 2024-09-18T14:41:31.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:31.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj710 datafile 2024-09-18T14:41:31.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:31.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj711 datafile 2024-09-18T14:41:31.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:31.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj712 datafile 2024-09-18T14:41:31.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:31.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj713 datafile 2024-09-18T14:41:31.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:31.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj714 datafile 2024-09-18T14:41:32.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:32.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj715 datafile 2024-09-18T14:41:32.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:32.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj716 datafile 2024-09-18T14:41:32.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:32.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj717 datafile 2024-09-18T14:41:32.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:32.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj718 datafile 2024-09-18T14:41:32.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:32.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj719 datafile 2024-09-18T14:41:32.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:32.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj720 datafile 2024-09-18T14:41:32.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:32.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj721 datafile 2024-09-18T14:41:33.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:33.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj722 datafile 2024-09-18T14:41:33.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:33.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj723 datafile 2024-09-18T14:41:33.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:33.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj724 datafile 2024-09-18T14:41:33.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:33.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj725 datafile 2024-09-18T14:41:33.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:33.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj726 datafile 2024-09-18T14:41:33.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:33.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj727 datafile 2024-09-18T14:41:33.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:33.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj728 datafile 2024-09-18T14:41:34.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:34.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj729 datafile 2024-09-18T14:41:34.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:34.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj730 datafile 2024-09-18T14:41:34.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:34.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj731 datafile 2024-09-18T14:41:34.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:34.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj732 datafile 2024-09-18T14:41:34.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:34.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj733 datafile 2024-09-18T14:41:34.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:34.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj734 datafile 2024-09-18T14:41:35.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:35.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj735 datafile 2024-09-18T14:41:35.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:35.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj736 datafile 2024-09-18T14:41:35.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:35.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj737 datafile 2024-09-18T14:41:35.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:35.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj738 datafile 2024-09-18T14:41:35.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:35.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj739 datafile 2024-09-18T14:41:35.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:35.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj740 datafile 2024-09-18T14:41:35.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:35.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj741 datafile 2024-09-18T14:41:36.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:36.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj742 datafile 2024-09-18T14:41:36.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:36.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj743 datafile 2024-09-18T14:41:36.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:36.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj744 datafile 2024-09-18T14:41:36.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:36.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj745 datafile 2024-09-18T14:41:36.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:36.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj746 datafile 2024-09-18T14:41:36.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:36.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj747 datafile 2024-09-18T14:41:37.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:37.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj748 datafile 2024-09-18T14:41:37.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:37.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj749 datafile 2024-09-18T14:41:37.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:37.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj750 datafile 2024-09-18T14:41:37.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:37.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj751 datafile 2024-09-18T14:41:37.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:37.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj752 datafile 2024-09-18T14:41:38.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:38.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj753 datafile 2024-09-18T14:41:38.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:38.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj754 datafile 2024-09-18T14:41:38.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:38.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj755 datafile 2024-09-18T14:41:38.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:38.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj756 datafile 2024-09-18T14:41:38.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:38.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj757 datafile 2024-09-18T14:41:38.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:38.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj758 datafile 2024-09-18T14:41:39.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:39.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj759 datafile 2024-09-18T14:41:39.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:39.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj760 datafile 2024-09-18T14:41:39.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:39.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj761 datafile 2024-09-18T14:41:39.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:39.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj762 datafile 2024-09-18T14:41:39.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:39.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj763 datafile 2024-09-18T14:41:39.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:39.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj764 datafile 2024-09-18T14:41:40.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:40.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj765 datafile 2024-09-18T14:41:40.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:40.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj766 datafile 2024-09-18T14:41:40.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:40.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj767 datafile 2024-09-18T14:41:40.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:40.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj768 datafile 2024-09-18T14:41:40.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:40.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj769 datafile 2024-09-18T14:41:40.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:40.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj770 datafile 2024-09-18T14:41:40.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:40.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj771 datafile 2024-09-18T14:41:41.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:41.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj772 datafile 2024-09-18T14:41:41.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:41.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj773 datafile 2024-09-18T14:41:41.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:41.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj774 datafile 2024-09-18T14:41:41.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:41.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj775 datafile 2024-09-18T14:41:41.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:41.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj776 datafile 2024-09-18T14:41:41.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:41.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj777 datafile 2024-09-18T14:41:41.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:41.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj778 datafile 2024-09-18T14:41:42.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:42.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj779 datafile 2024-09-18T14:41:42.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:42.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj780 datafile 2024-09-18T14:41:42.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:42.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj781 datafile 2024-09-18T14:41:42.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:42.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj782 datafile 2024-09-18T14:41:42.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:42.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj783 datafile 2024-09-18T14:41:42.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:42.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj784 datafile 2024-09-18T14:41:43.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:43.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj785 datafile 2024-09-18T14:41:43.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:43.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj786 datafile 2024-09-18T14:41:43.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:43.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj787 datafile 2024-09-18T14:41:43.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:43.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj788 datafile 2024-09-18T14:41:43.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:43.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj789 datafile 2024-09-18T14:41:43.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:43.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj790 datafile 2024-09-18T14:41:43.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:43.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj791 datafile 2024-09-18T14:41:44.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:44.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj792 datafile 2024-09-18T14:41:44.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:44.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj793 datafile 2024-09-18T14:41:44.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:44.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj794 datafile 2024-09-18T14:41:44.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:44.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj795 datafile 2024-09-18T14:41:44.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:44.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj796 datafile 2024-09-18T14:41:44.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:44.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj797 datafile 2024-09-18T14:41:44.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:44.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj798 datafile 2024-09-18T14:41:45.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:45.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj799 datafile 2024-09-18T14:41:45.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:45.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj800 datafile 2024-09-18T14:41:45.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:45.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj801 datafile 2024-09-18T14:41:45.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:45.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj802 datafile 2024-09-18T14:41:45.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:45.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj803 datafile 2024-09-18T14:41:45.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:45.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj804 datafile 2024-09-18T14:41:46.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:46.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj805 datafile 2024-09-18T14:41:46.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:46.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj806 datafile 2024-09-18T14:41:46.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:46.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj807 datafile 2024-09-18T14:41:46.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:46.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj808 datafile 2024-09-18T14:41:46.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:46.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj809 datafile 2024-09-18T14:41:47.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:47.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj810 datafile 2024-09-18T14:41:47.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:47.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj811 datafile 2024-09-18T14:41:47.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:47.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj812 datafile 2024-09-18T14:41:47.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:47.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj813 datafile 2024-09-18T14:41:47.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:47.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj814 datafile 2024-09-18T14:41:47.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:47.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj815 datafile 2024-09-18T14:41:48.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:48.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj816 datafile 2024-09-18T14:41:48.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:48.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj817 datafile 2024-09-18T14:41:48.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:48.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj818 datafile 2024-09-18T14:41:48.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:48.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj819 datafile 2024-09-18T14:41:48.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:48.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj820 datafile 2024-09-18T14:41:48.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:48.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj821 datafile 2024-09-18T14:41:49.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:49.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj822 datafile 2024-09-18T14:41:49.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:49.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj823 datafile 2024-09-18T14:41:49.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:49.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj824 datafile 2024-09-18T14:41:49.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:49.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj825 datafile 2024-09-18T14:41:49.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:49.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj826 datafile 2024-09-18T14:41:49.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:49.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj827 datafile 2024-09-18T14:41:49.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:49.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj828 datafile 2024-09-18T14:41:50.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:50.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj829 datafile 2024-09-18T14:41:50.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:50.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj830 datafile 2024-09-18T14:41:50.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:50.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj831 datafile 2024-09-18T14:41:50.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:50.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj832 datafile 2024-09-18T14:41:50.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:50.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj833 datafile 2024-09-18T14:41:50.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:50.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj834 datafile 2024-09-18T14:41:50.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:50.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj835 datafile 2024-09-18T14:41:51.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:51.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj836 datafile 2024-09-18T14:41:51.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:51.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj837 datafile 2024-09-18T14:41:51.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:51.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj838 datafile 2024-09-18T14:41:51.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:51.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj839 datafile 2024-09-18T14:41:51.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:51.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj840 datafile 2024-09-18T14:41:51.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:51.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj841 datafile 2024-09-18T14:41:52.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:52.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj842 datafile 2024-09-18T14:41:52.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:52.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj843 datafile 2024-09-18T14:41:52.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:52.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj844 datafile 2024-09-18T14:41:52.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:52.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj845 datafile 2024-09-18T14:41:52.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:52.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj846 datafile 2024-09-18T14:41:52.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:52.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj847 datafile 2024-09-18T14:41:52.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:52.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj848 datafile 2024-09-18T14:41:53.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:53.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj849 datafile 2024-09-18T14:41:53.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:53.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj850 datafile 2024-09-18T14:41:53.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:53.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj851 datafile 2024-09-18T14:41:53.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:53.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj852 datafile 2024-09-18T14:41:53.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:53.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj853 datafile 2024-09-18T14:41:53.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:53.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj854 datafile 2024-09-18T14:41:54.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:54.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj855 datafile 2024-09-18T14:41:54.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:54.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj856 datafile 2024-09-18T14:41:54.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:54.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj857 datafile 2024-09-18T14:41:54.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:54.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj858 datafile 2024-09-18T14:41:54.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:54.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj859 datafile 2024-09-18T14:41:54.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:54.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj860 datafile 2024-09-18T14:41:55.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:55.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj861 datafile 2024-09-18T14:41:55.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:55.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj862 datafile 2024-09-18T14:41:55.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:55.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj863 datafile 2024-09-18T14:41:55.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:55.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj864 datafile 2024-09-18T14:41:55.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:55.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj865 datafile 2024-09-18T14:41:55.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:55.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj866 datafile 2024-09-18T14:41:56.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:56.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj867 datafile 2024-09-18T14:41:56.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:56.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj868 datafile 2024-09-18T14:41:56.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:56.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj869 datafile 2024-09-18T14:41:56.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:56.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj870 datafile 2024-09-18T14:41:56.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:56.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj871 datafile 2024-09-18T14:41:56.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:56.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj872 datafile 2024-09-18T14:41:57.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:57.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj873 datafile 2024-09-18T14:41:57.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:57.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj874 datafile 2024-09-18T14:41:57.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:57.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj875 datafile 2024-09-18T14:41:57.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:57.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj876 datafile 2024-09-18T14:41:57.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:57.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj877 datafile 2024-09-18T14:41:58.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:58.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj878 datafile 2024-09-18T14:41:58.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:58.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj879 datafile 2024-09-18T14:41:58.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:58.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj880 datafile 2024-09-18T14:41:58.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:58.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj881 datafile 2024-09-18T14:41:58.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:58.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj882 datafile 2024-09-18T14:41:58.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:58.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj883 datafile 2024-09-18T14:41:58.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:58.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj884 datafile 2024-09-18T14:41:59.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:59.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj885 datafile 2024-09-18T14:41:59.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:59.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj886 datafile 2024-09-18T14:41:59.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:59.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj887 datafile 2024-09-18T14:41:59.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:59.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj888 datafile 2024-09-18T14:41:59.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:59.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj889 datafile 2024-09-18T14:41:59.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:41:59.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj890 datafile 2024-09-18T14:42:00.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:00.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj891 datafile 2024-09-18T14:42:00.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:00.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj892 datafile 2024-09-18T14:42:00.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:00.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj893 datafile 2024-09-18T14:42:00.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:00.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj894 datafile 2024-09-18T14:42:00.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:00.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj895 datafile 2024-09-18T14:42:00.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:00.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj896 datafile 2024-09-18T14:42:01.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:01.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj897 datafile 2024-09-18T14:42:01.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:01.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj898 datafile 2024-09-18T14:42:01.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:01.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj899 datafile 2024-09-18T14:42:01.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:01.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj900 datafile 2024-09-18T14:42:01.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:01.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj901 datafile 2024-09-18T14:42:01.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:01.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj902 datafile 2024-09-18T14:42:02.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:02.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj903 datafile 2024-09-18T14:42:02.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:02.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj904 datafile 2024-09-18T14:42:02.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:02.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj905 datafile 2024-09-18T14:42:02.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:02.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj906 datafile 2024-09-18T14:42:02.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:02.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj907 datafile 2024-09-18T14:42:02.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:02.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj908 datafile 2024-09-18T14:42:03.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:03.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj909 datafile 2024-09-18T14:42:03.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:03.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj910 datafile 2024-09-18T14:42:03.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:03.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj911 datafile 2024-09-18T14:42:03.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:03.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj912 datafile 2024-09-18T14:42:03.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:03.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj913 datafile 2024-09-18T14:42:03.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:03.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj914 datafile 2024-09-18T14:42:04.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:04.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj915 datafile 2024-09-18T14:42:04.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:04.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj916 datafile 2024-09-18T14:42:04.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:04.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj917 datafile 2024-09-18T14:42:04.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:04.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj918 datafile 2024-09-18T14:42:04.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:04.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj919 datafile 2024-09-18T14:42:04.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:04.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj920 datafile 2024-09-18T14:42:05.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:05.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj921 datafile 2024-09-18T14:42:05.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:05.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj922 datafile 2024-09-18T14:42:05.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:05.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj923 datafile 2024-09-18T14:42:05.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:05.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj924 datafile 2024-09-18T14:42:05.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:05.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj925 datafile 2024-09-18T14:42:05.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:05.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj926 datafile 2024-09-18T14:42:06.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:06.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj927 datafile 2024-09-18T14:42:06.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:06.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj928 datafile 2024-09-18T14:42:06.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:06.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj929 datafile 2024-09-18T14:42:06.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:06.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj930 datafile 2024-09-18T14:42:06.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:06.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj931 datafile 2024-09-18T14:42:06.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:06.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj932 datafile 2024-09-18T14:42:07.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:07.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj933 datafile 2024-09-18T14:42:07.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:07.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj934 datafile 2024-09-18T14:42:07.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:07.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj935 datafile 2024-09-18T14:42:07.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:07.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj936 datafile 2024-09-18T14:42:07.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:07.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj937 datafile 2024-09-18T14:42:07.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:07.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj938 datafile 2024-09-18T14:42:08.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:08.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj939 datafile 2024-09-18T14:42:08.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:08.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj940 datafile 2024-09-18T14:42:08.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:08.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj941 datafile 2024-09-18T14:42:08.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:08.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj942 datafile 2024-09-18T14:42:08.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:08.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj943 datafile 2024-09-18T14:42:08.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:08.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj944 datafile 2024-09-18T14:42:09.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:09.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj945 datafile 2024-09-18T14:42:09.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:09.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj946 datafile 2024-09-18T14:42:09.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:09.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj947 datafile 2024-09-18T14:42:09.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:09.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj948 datafile 2024-09-18T14:42:09.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:09.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj949 datafile 2024-09-18T14:42:09.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:09.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj950 datafile 2024-09-18T14:42:10.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:10.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj951 datafile 2024-09-18T14:42:10.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:10.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj952 datafile 2024-09-18T14:42:10.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:10.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj953 datafile 2024-09-18T14:42:10.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:10.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj954 datafile 2024-09-18T14:42:10.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:10.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj955 datafile 2024-09-18T14:42:10.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:10.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj956 datafile 2024-09-18T14:42:11.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:11.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj957 datafile 2024-09-18T14:42:11.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:11.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj958 datafile 2024-09-18T14:42:11.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:11.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj959 datafile 2024-09-18T14:42:11.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:11.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj960 datafile 2024-09-18T14:42:11.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:11.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj961 datafile 2024-09-18T14:42:11.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:11.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj962 datafile 2024-09-18T14:42:11.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:11.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj963 datafile 2024-09-18T14:42:12.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:12.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj964 datafile 2024-09-18T14:42:12.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:12.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj965 datafile 2024-09-18T14:42:12.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:12.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj966 datafile 2024-09-18T14:42:12.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:12.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj967 datafile 2024-09-18T14:42:12.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:12.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj968 datafile 2024-09-18T14:42:12.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:12.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj969 datafile 2024-09-18T14:42:13.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:13.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj970 datafile 2024-09-18T14:42:13.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:13.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj971 datafile 2024-09-18T14:42:13.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:13.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj972 datafile 2024-09-18T14:42:13.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:13.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj973 datafile 2024-09-18T14:42:13.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:13.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj974 datafile 2024-09-18T14:42:13.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:13.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj975 datafile 2024-09-18T14:42:14.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:14.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj976 datafile 2024-09-18T14:42:14.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:14.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj977 datafile 2024-09-18T14:42:14.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:14.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj978 datafile 2024-09-18T14:42:14.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:14.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj979 datafile 2024-09-18T14:42:14.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:14.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj980 datafile 2024-09-18T14:42:14.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:14.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj981 datafile 2024-09-18T14:42:15.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:15.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj982 datafile 2024-09-18T14:42:15.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:15.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj983 datafile 2024-09-18T14:42:15.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:15.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj984 datafile 2024-09-18T14:42:15.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:15.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj985 datafile 2024-09-18T14:42:15.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:15.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj986 datafile 2024-09-18T14:42:15.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:15.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj987 datafile 2024-09-18T14:42:16.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:16.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj988 datafile 2024-09-18T14:42:16.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:16.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj989 datafile 2024-09-18T14:42:16.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:16.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj990 datafile 2024-09-18T14:42:16.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:16.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj991 datafile 2024-09-18T14:42:16.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:16.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj992 datafile 2024-09-18T14:42:16.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:16.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj993 datafile 2024-09-18T14:42:16.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:16.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj994 datafile 2024-09-18T14:42:17.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:17.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj995 datafile 2024-09-18T14:42:17.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:17.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj996 datafile 2024-09-18T14:42:17.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:17.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj997 datafile 2024-09-18T14:42:17.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:17.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj998 datafile 2024-09-18T14:42:18.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:18.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj999 datafile 2024-09-18T14:42:18.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:18.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test2 put obj1000 datafile 2024-09-18T14:42:18.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for i in $(seq 1 $POOLS) 2024-09-18T14:42:18.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: seq 1 1000 2024-09-18T14:42:18.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:18.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj1 datafile 2024-09-18T14:42:18.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:18.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj2 datafile 2024-09-18T14:42:18.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:18.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj3 datafile 2024-09-18T14:42:18.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:18.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj4 datafile 2024-09-18T14:42:19.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:19.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj5 datafile 2024-09-18T14:42:19.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:19.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj6 datafile 2024-09-18T14:42:19.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:19.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj7 datafile 2024-09-18T14:42:19.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:19.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj8 datafile 2024-09-18T14:42:19.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:19.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj9 datafile 2024-09-18T14:42:19.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:19.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj10 datafile 2024-09-18T14:42:20.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:20.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj11 datafile 2024-09-18T14:42:20.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:20.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj12 datafile 2024-09-18T14:42:20.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:20.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj13 datafile 2024-09-18T14:42:20.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:20.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj14 datafile 2024-09-18T14:42:20.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:20.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj15 datafile 2024-09-18T14:42:20.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:20.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj16 datafile 2024-09-18T14:42:20.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:20.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj17 datafile 2024-09-18T14:42:21.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:21.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj18 datafile 2024-09-18T14:42:21.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:21.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj19 datafile 2024-09-18T14:42:21.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:21.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj20 datafile 2024-09-18T14:42:21.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:21.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj21 datafile 2024-09-18T14:42:21.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:21.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj22 datafile 2024-09-18T14:42:21.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:21.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj23 datafile 2024-09-18T14:42:22.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:22.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj24 datafile 2024-09-18T14:42:22.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:22.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj25 datafile 2024-09-18T14:42:22.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:22.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj26 datafile 2024-09-18T14:42:22.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:22.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj27 datafile 2024-09-18T14:42:22.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:22.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj28 datafile 2024-09-18T14:42:22.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:22.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj29 datafile 2024-09-18T14:42:22.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:22.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj30 datafile 2024-09-18T14:42:23.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:23.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj31 datafile 2024-09-18T14:42:23.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:23.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj32 datafile 2024-09-18T14:42:23.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:23.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj33 datafile 2024-09-18T14:42:23.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:23.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj34 datafile 2024-09-18T14:42:23.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:23.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj35 datafile 2024-09-18T14:42:24.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:24.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj36 datafile 2024-09-18T14:42:24.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:24.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj37 datafile 2024-09-18T14:42:24.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:24.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj38 datafile 2024-09-18T14:42:24.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:24.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj39 datafile 2024-09-18T14:42:24.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:24.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj40 datafile 2024-09-18T14:42:24.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:24.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj41 datafile 2024-09-18T14:42:24.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:24.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj42 datafile 2024-09-18T14:42:25.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:25.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj43 datafile 2024-09-18T14:42:25.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:25.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj44 datafile 2024-09-18T14:42:25.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:25.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj45 datafile 2024-09-18T14:42:25.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:25.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj46 datafile 2024-09-18T14:42:25.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:25.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj47 datafile 2024-09-18T14:42:25.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:25.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj48 datafile 2024-09-18T14:42:25.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:25.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj49 datafile 2024-09-18T14:42:26.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:26.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj50 datafile 2024-09-18T14:42:26.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:26.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj51 datafile 2024-09-18T14:42:26.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:26.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj52 datafile 2024-09-18T14:42:26.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:26.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj53 datafile 2024-09-18T14:42:26.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:26.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj54 datafile 2024-09-18T14:42:26.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:26.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj55 datafile 2024-09-18T14:42:27.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:27.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj56 datafile 2024-09-18T14:42:27.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:27.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj57 datafile 2024-09-18T14:42:27.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:27.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj58 datafile 2024-09-18T14:42:27.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:27.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj59 datafile 2024-09-18T14:42:27.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:27.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj60 datafile 2024-09-18T14:42:27.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:27.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj61 datafile 2024-09-18T14:42:28.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:28.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj62 datafile 2024-09-18T14:42:28.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:28.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj63 datafile 2024-09-18T14:42:28.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:28.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj64 datafile 2024-09-18T14:42:28.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:28.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj65 datafile 2024-09-18T14:42:28.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:28.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj66 datafile 2024-09-18T14:42:29.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:29.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj67 datafile 2024-09-18T14:42:29.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:29.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj68 datafile 2024-09-18T14:42:29.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:29.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj69 datafile 2024-09-18T14:42:29.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:29.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj70 datafile 2024-09-18T14:42:29.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:29.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj71 datafile 2024-09-18T14:42:29.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:29.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj72 datafile 2024-09-18T14:42:30.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:30.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj73 datafile 2024-09-18T14:42:30.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:30.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj74 datafile 2024-09-18T14:42:30.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:30.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj75 datafile 2024-09-18T14:42:30.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:30.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj76 datafile 2024-09-18T14:42:30.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:30.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj77 datafile 2024-09-18T14:42:30.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:30.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj78 datafile 2024-09-18T14:42:31.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:31.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj79 datafile 2024-09-18T14:42:31.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:31.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj80 datafile 2024-09-18T14:42:31.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:31.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj81 datafile 2024-09-18T14:42:31.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:31.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj82 datafile 2024-09-18T14:42:31.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:31.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj83 datafile 2024-09-18T14:42:31.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:31.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj84 datafile 2024-09-18T14:42:31.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:31.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj85 datafile 2024-09-18T14:42:32.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:32.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj86 datafile 2024-09-18T14:42:32.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:32.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj87 datafile 2024-09-18T14:42:32.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:32.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj88 datafile 2024-09-18T14:42:32.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:32.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj89 datafile 2024-09-18T14:42:32.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:32.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj90 datafile 2024-09-18T14:42:32.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:32.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj91 datafile 2024-09-18T14:42:33.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:33.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj92 datafile 2024-09-18T14:42:33.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:33.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj93 datafile 2024-09-18T14:42:33.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:33.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj94 datafile 2024-09-18T14:42:33.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:33.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj95 datafile 2024-09-18T14:42:33.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:33.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj96 datafile 2024-09-18T14:42:33.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:33.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj97 datafile 2024-09-18T14:42:34.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:34.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj98 datafile 2024-09-18T14:42:34.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:34.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj99 datafile 2024-09-18T14:42:34.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:34.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj100 datafile 2024-09-18T14:42:34.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:34.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj101 datafile 2024-09-18T14:42:34.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:34.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj102 datafile 2024-09-18T14:42:35.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:35.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj103 datafile 2024-09-18T14:42:35.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:35.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj104 datafile 2024-09-18T14:42:35.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:35.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj105 datafile 2024-09-18T14:42:35.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:35.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj106 datafile 2024-09-18T14:42:35.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:35.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj107 datafile 2024-09-18T14:42:35.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:35.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj108 datafile 2024-09-18T14:42:36.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:36.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj109 datafile 2024-09-18T14:42:36.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:36.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj110 datafile 2024-09-18T14:42:36.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:36.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj111 datafile 2024-09-18T14:42:36.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:36.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj112 datafile 2024-09-18T14:42:36.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:36.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj113 datafile 2024-09-18T14:42:36.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:36.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj114 datafile 2024-09-18T14:42:37.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:37.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj115 datafile 2024-09-18T14:42:37.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:37.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj116 datafile 2024-09-18T14:42:37.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:37.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj117 datafile 2024-09-18T14:42:37.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:37.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj118 datafile 2024-09-18T14:42:37.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:37.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj119 datafile 2024-09-18T14:42:37.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:37.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj120 datafile 2024-09-18T14:42:38.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:38.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj121 datafile 2024-09-18T14:42:38.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:38.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj122 datafile 2024-09-18T14:42:38.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:38.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj123 datafile 2024-09-18T14:42:38.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:38.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj124 datafile 2024-09-18T14:42:38.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:38.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj125 datafile 2024-09-18T14:42:38.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:38.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj126 datafile 2024-09-18T14:42:39.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:39.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj127 datafile 2024-09-18T14:42:39.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:39.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj128 datafile 2024-09-18T14:42:39.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:39.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj129 datafile 2024-09-18T14:42:39.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:39.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj130 datafile 2024-09-18T14:42:39.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:39.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj131 datafile 2024-09-18T14:42:39.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:39.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj132 datafile 2024-09-18T14:42:40.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:40.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj133 datafile 2024-09-18T14:42:40.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:40.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj134 datafile 2024-09-18T14:42:40.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:40.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj135 datafile 2024-09-18T14:42:40.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:40.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj136 datafile 2024-09-18T14:42:40.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:40.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj137 datafile 2024-09-18T14:42:40.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:40.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj138 datafile 2024-09-18T14:42:41.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:41.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj139 datafile 2024-09-18T14:42:41.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:41.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj140 datafile 2024-09-18T14:42:41.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:41.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj141 datafile 2024-09-18T14:42:41.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:41.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj142 datafile 2024-09-18T14:42:41.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:41.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj143 datafile 2024-09-18T14:42:41.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:41.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj144 datafile 2024-09-18T14:42:42.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:42.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj145 datafile 2024-09-18T14:42:42.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:42.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj146 datafile 2024-09-18T14:42:42.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:42.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj147 datafile 2024-09-18T14:42:42.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:42.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj148 datafile 2024-09-18T14:42:42.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:42.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj149 datafile 2024-09-18T14:42:42.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:42.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj150 datafile 2024-09-18T14:42:43.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:43.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj151 datafile 2024-09-18T14:42:43.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:43.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj152 datafile 2024-09-18T14:42:43.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:43.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj153 datafile 2024-09-18T14:42:43.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:43.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj154 datafile 2024-09-18T14:42:43.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:43.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj155 datafile 2024-09-18T14:42:43.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:43.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj156 datafile 2024-09-18T14:42:44.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:44.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj157 datafile 2024-09-18T14:42:44.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:44.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj158 datafile 2024-09-18T14:42:44.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:44.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj159 datafile 2024-09-18T14:42:44.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:44.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj160 datafile 2024-09-18T14:42:44.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:44.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj161 datafile 2024-09-18T14:42:44.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:44.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj162 datafile 2024-09-18T14:42:44.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:44.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj163 datafile 2024-09-18T14:42:45.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:45.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj164 datafile 2024-09-18T14:42:45.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:45.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj165 datafile 2024-09-18T14:42:45.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:45.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj166 datafile 2024-09-18T14:42:45.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:45.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj167 datafile 2024-09-18T14:42:45.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:45.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj168 datafile 2024-09-18T14:42:45.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:45.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj169 datafile 2024-09-18T14:42:46.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:46.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj170 datafile 2024-09-18T14:42:46.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:46.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj171 datafile 2024-09-18T14:42:46.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:46.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj172 datafile 2024-09-18T14:42:46.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:46.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj173 datafile 2024-09-18T14:42:46.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:46.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj174 datafile 2024-09-18T14:42:46.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:46.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj175 datafile 2024-09-18T14:42:47.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:47.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj176 datafile 2024-09-18T14:42:47.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:47.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj177 datafile 2024-09-18T14:42:47.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:47.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj178 datafile 2024-09-18T14:42:47.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:47.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj179 datafile 2024-09-18T14:42:47.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:47.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj180 datafile 2024-09-18T14:42:47.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:47.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj181 datafile 2024-09-18T14:42:48.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:48.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj182 datafile 2024-09-18T14:42:48.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:48.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj183 datafile 2024-09-18T14:42:48.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:48.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj184 datafile 2024-09-18T14:42:48.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:48.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj185 datafile 2024-09-18T14:42:48.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:48.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj186 datafile 2024-09-18T14:42:49.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:49.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj187 datafile 2024-09-18T14:42:49.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:49.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj188 datafile 2024-09-18T14:42:49.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:49.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj189 datafile 2024-09-18T14:42:49.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:49.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj190 datafile 2024-09-18T14:42:49.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:49.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj191 datafile 2024-09-18T14:42:49.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:49.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj192 datafile 2024-09-18T14:42:50.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:50.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj193 datafile 2024-09-18T14:42:50.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:50.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj194 datafile 2024-09-18T14:42:50.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:50.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj195 datafile 2024-09-18T14:42:50.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:50.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj196 datafile 2024-09-18T14:42:50.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:50.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj197 datafile 2024-09-18T14:42:50.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:50.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj198 datafile 2024-09-18T14:42:50.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:50.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj199 datafile 2024-09-18T14:42:51.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:51.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj200 datafile 2024-09-18T14:42:51.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:51.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj201 datafile 2024-09-18T14:42:51.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:51.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj202 datafile 2024-09-18T14:42:51.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:51.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj203 datafile 2024-09-18T14:42:51.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:51.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj204 datafile 2024-09-18T14:42:51.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:51.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj205 datafile 2024-09-18T14:42:52.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:52.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj206 datafile 2024-09-18T14:42:52.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:52.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj207 datafile 2024-09-18T14:42:52.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:52.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj208 datafile 2024-09-18T14:42:52.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:52.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj209 datafile 2024-09-18T14:42:52.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:52.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj210 datafile 2024-09-18T14:42:52.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:52.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj211 datafile 2024-09-18T14:42:53.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:53.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj212 datafile 2024-09-18T14:42:53.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:53.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj213 datafile 2024-09-18T14:42:53.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:53.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj214 datafile 2024-09-18T14:42:53.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:53.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj215 datafile 2024-09-18T14:42:53.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:53.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj216 datafile 2024-09-18T14:42:53.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:53.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj217 datafile 2024-09-18T14:42:54.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:54.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj218 datafile 2024-09-18T14:42:54.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:54.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj219 datafile 2024-09-18T14:42:54.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:54.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj220 datafile 2024-09-18T14:42:54.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:54.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj221 datafile 2024-09-18T14:42:54.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:54.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj222 datafile 2024-09-18T14:42:54.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:54.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj223 datafile 2024-09-18T14:42:55.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:55.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj224 datafile 2024-09-18T14:42:55.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:55.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj225 datafile 2024-09-18T14:42:55.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:55.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj226 datafile 2024-09-18T14:42:55.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:55.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj227 datafile 2024-09-18T14:42:55.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:55.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj228 datafile 2024-09-18T14:42:55.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:55.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj229 datafile 2024-09-18T14:42:56.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:56.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj230 datafile 2024-09-18T14:42:56.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:56.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj231 datafile 2024-09-18T14:42:56.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:56.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj232 datafile 2024-09-18T14:42:56.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:56.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj233 datafile 2024-09-18T14:42:56.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:56.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj234 datafile 2024-09-18T14:42:56.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:56.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj235 datafile 2024-09-18T14:42:57.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:57.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj236 datafile 2024-09-18T14:42:57.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:57.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj237 datafile 2024-09-18T14:42:57.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:57.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj238 datafile 2024-09-18T14:42:57.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:57.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj239 datafile 2024-09-18T14:42:57.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:57.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj240 datafile 2024-09-18T14:42:57.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:57.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj241 datafile 2024-09-18T14:42:58.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:58.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj242 datafile 2024-09-18T14:42:58.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:58.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj243 datafile 2024-09-18T14:42:58.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:58.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj244 datafile 2024-09-18T14:42:58.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:58.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj245 datafile 2024-09-18T14:42:58.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:58.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj246 datafile 2024-09-18T14:42:58.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:58.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj247 datafile 2024-09-18T14:42:59.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:59.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj248 datafile 2024-09-18T14:42:59.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:59.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj249 datafile 2024-09-18T14:42:59.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:59.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj250 datafile 2024-09-18T14:42:59.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:59.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj251 datafile 2024-09-18T14:42:59.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:42:59.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj252 datafile 2024-09-18T14:43:00.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:00.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj253 datafile 2024-09-18T14:43:00.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:00.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj254 datafile 2024-09-18T14:43:00.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:00.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj255 datafile 2024-09-18T14:43:00.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:00.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj256 datafile 2024-09-18T14:43:00.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:00.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj257 datafile 2024-09-18T14:43:00.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:00.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj258 datafile 2024-09-18T14:43:01.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:01.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj259 datafile 2024-09-18T14:43:01.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:01.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj260 datafile 2024-09-18T14:43:01.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:01.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj261 datafile 2024-09-18T14:43:01.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:01.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj262 datafile 2024-09-18T14:43:01.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:01.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj263 datafile 2024-09-18T14:43:01.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:02.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj264 datafile 2024-09-18T14:43:02.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:02.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj265 datafile 2024-09-18T14:43:02.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:02.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj266 datafile 2024-09-18T14:43:02.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:02.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj267 datafile 2024-09-18T14:43:02.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:02.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj268 datafile 2024-09-18T14:43:02.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:02.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj269 datafile 2024-09-18T14:43:02.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:02.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj270 datafile 2024-09-18T14:43:03.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:03.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj271 datafile 2024-09-18T14:43:03.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:03.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj272 datafile 2024-09-18T14:43:03.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:03.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj273 datafile 2024-09-18T14:43:03.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:03.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj274 datafile 2024-09-18T14:43:03.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:03.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj275 datafile 2024-09-18T14:43:03.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:03.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj276 datafile 2024-09-18T14:43:04.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:04.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj277 datafile 2024-09-18T14:43:04.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:04.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj278 datafile 2024-09-18T14:43:04.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:04.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj279 datafile 2024-09-18T14:43:04.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:04.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj280 datafile 2024-09-18T14:43:04.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:04.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj281 datafile 2024-09-18T14:43:04.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:04.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj282 datafile 2024-09-18T14:43:05.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:05.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj283 datafile 2024-09-18T14:43:05.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:05.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj284 datafile 2024-09-18T14:43:05.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:05.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj285 datafile 2024-09-18T14:43:05.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:05.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj286 datafile 2024-09-18T14:43:05.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:05.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj287 datafile 2024-09-18T14:43:05.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:05.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj288 datafile 2024-09-18T14:43:05.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:05.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj289 datafile 2024-09-18T14:43:06.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:06.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj290 datafile 2024-09-18T14:43:06.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:06.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj291 datafile 2024-09-18T14:43:06.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:06.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj292 datafile 2024-09-18T14:43:06.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:06.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj293 datafile 2024-09-18T14:43:06.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:06.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj294 datafile 2024-09-18T14:43:06.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:06.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj295 datafile 2024-09-18T14:43:07.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:07.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj296 datafile 2024-09-18T14:43:07.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:07.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj297 datafile 2024-09-18T14:43:07.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:07.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj298 datafile 2024-09-18T14:43:07.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:07.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj299 datafile 2024-09-18T14:43:07.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:07.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj300 datafile 2024-09-18T14:43:07.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:07.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj301 datafile 2024-09-18T14:43:08.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:08.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj302 datafile 2024-09-18T14:43:08.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:08.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj303 datafile 2024-09-18T14:43:08.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:08.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj304 datafile 2024-09-18T14:43:08.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:08.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj305 datafile 2024-09-18T14:43:08.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:08.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj306 datafile 2024-09-18T14:43:08.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:08.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj307 datafile 2024-09-18T14:43:09.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:09.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj308 datafile 2024-09-18T14:43:09.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:09.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj309 datafile 2024-09-18T14:43:09.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:09.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj310 datafile 2024-09-18T14:43:09.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:09.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj311 datafile 2024-09-18T14:43:09.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:09.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj312 datafile 2024-09-18T14:43:09.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:09.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj313 datafile 2024-09-18T14:43:09.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:09.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj314 datafile 2024-09-18T14:43:10.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:10.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj315 datafile 2024-09-18T14:43:10.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:10.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj316 datafile 2024-09-18T14:43:10.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:10.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj317 datafile 2024-09-18T14:43:10.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:10.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj318 datafile 2024-09-18T14:43:10.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:10.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj319 datafile 2024-09-18T14:43:10.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:10.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj320 datafile 2024-09-18T14:43:11.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:11.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj321 datafile 2024-09-18T14:43:11.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:11.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj322 datafile 2024-09-18T14:43:11.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:11.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj323 datafile 2024-09-18T14:43:11.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:11.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj324 datafile 2024-09-18T14:43:11.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:11.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj325 datafile 2024-09-18T14:43:11.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:11.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj326 datafile 2024-09-18T14:43:11.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:12.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj327 datafile 2024-09-18T14:43:12.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:12.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj328 datafile 2024-09-18T14:43:12.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:12.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj329 datafile 2024-09-18T14:43:12.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:12.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj330 datafile 2024-09-18T14:43:12.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:12.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj331 datafile 2024-09-18T14:43:12.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:12.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj332 datafile 2024-09-18T14:43:13.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:13.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj333 datafile 2024-09-18T14:43:13.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:13.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj334 datafile 2024-09-18T14:43:13.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:13.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj335 datafile 2024-09-18T14:43:13.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:13.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj336 datafile 2024-09-18T14:43:13.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:13.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj337 datafile 2024-09-18T14:43:13.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:13.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj338 datafile 2024-09-18T14:43:13.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:13.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj339 datafile 2024-09-18T14:43:14.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:14.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj340 datafile 2024-09-18T14:43:14.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:14.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj341 datafile 2024-09-18T14:43:14.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:14.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj342 datafile 2024-09-18T14:43:14.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:14.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj343 datafile 2024-09-18T14:43:14.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:14.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj344 datafile 2024-09-18T14:43:15.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:15.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj345 datafile 2024-09-18T14:43:15.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:15.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj346 datafile 2024-09-18T14:43:15.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:15.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj347 datafile 2024-09-18T14:43:15.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:15.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj348 datafile 2024-09-18T14:43:15.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:15.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj349 datafile 2024-09-18T14:43:15.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:15.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj350 datafile 2024-09-18T14:43:16.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:16.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj351 datafile 2024-09-18T14:43:16.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:16.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj352 datafile 2024-09-18T14:43:16.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:16.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj353 datafile 2024-09-18T14:43:16.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:16.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj354 datafile 2024-09-18T14:43:16.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:16.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj355 datafile 2024-09-18T14:43:16.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:16.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj356 datafile 2024-09-18T14:43:16.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:16.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj357 datafile 2024-09-18T14:43:17.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:17.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj358 datafile 2024-09-18T14:43:17.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:17.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj359 datafile 2024-09-18T14:43:17.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:17.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj360 datafile 2024-09-18T14:43:17.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:17.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj361 datafile 2024-09-18T14:43:17.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:17.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj362 datafile 2024-09-18T14:43:17.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:17.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj363 datafile 2024-09-18T14:43:18.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:18.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj364 datafile 2024-09-18T14:43:18.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:18.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj365 datafile 2024-09-18T14:43:18.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:18.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj366 datafile 2024-09-18T14:43:18.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:18.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj367 datafile 2024-09-18T14:43:18.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:18.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj368 datafile 2024-09-18T14:43:18.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:18.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj369 datafile 2024-09-18T14:43:19.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:19.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj370 datafile 2024-09-18T14:43:19.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:19.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj371 datafile 2024-09-18T14:43:19.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:19.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj372 datafile 2024-09-18T14:43:19.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:19.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj373 datafile 2024-09-18T14:43:19.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:19.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj374 datafile 2024-09-18T14:43:20.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:20.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj375 datafile 2024-09-18T14:43:20.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:20.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj376 datafile 2024-09-18T14:43:20.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:20.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj377 datafile 2024-09-18T14:43:20.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:20.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj378 datafile 2024-09-18T14:43:20.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:20.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj379 datafile 2024-09-18T14:43:20.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:20.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj380 datafile 2024-09-18T14:43:20.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:20.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj381 datafile 2024-09-18T14:43:21.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:21.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj382 datafile 2024-09-18T14:43:21.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:21.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj383 datafile 2024-09-18T14:43:21.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:21.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj384 datafile 2024-09-18T14:43:21.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:21.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj385 datafile 2024-09-18T14:43:21.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:21.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj386 datafile 2024-09-18T14:43:22.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:22.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj387 datafile 2024-09-18T14:43:22.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:22.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj388 datafile 2024-09-18T14:43:22.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:22.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj389 datafile 2024-09-18T14:43:22.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:22.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj390 datafile 2024-09-18T14:43:22.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:22.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj391 datafile 2024-09-18T14:43:22.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:22.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj392 datafile 2024-09-18T14:43:22.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:22.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj393 datafile 2024-09-18T14:43:23.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:23.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj394 datafile 2024-09-18T14:43:23.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:23.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj395 datafile 2024-09-18T14:43:23.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:23.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj396 datafile 2024-09-18T14:43:23.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:23.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj397 datafile 2024-09-18T14:43:23.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:23.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj398 datafile 2024-09-18T14:43:24.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:24.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj399 datafile 2024-09-18T14:43:24.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:24.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj400 datafile 2024-09-18T14:43:24.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:24.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj401 datafile 2024-09-18T14:43:24.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:24.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj402 datafile 2024-09-18T14:43:24.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:24.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj403 datafile 2024-09-18T14:43:24.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:24.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj404 datafile 2024-09-18T14:43:25.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:25.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj405 datafile 2024-09-18T14:43:25.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:25.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj406 datafile 2024-09-18T14:43:25.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:25.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj407 datafile 2024-09-18T14:43:25.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:25.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj408 datafile 2024-09-18T14:43:25.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:25.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj409 datafile 2024-09-18T14:43:25.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:25.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj410 datafile 2024-09-18T14:43:26.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:26.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj411 datafile 2024-09-18T14:43:26.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:26.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj412 datafile 2024-09-18T14:43:26.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:26.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj413 datafile 2024-09-18T14:43:26.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:26.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj414 datafile 2024-09-18T14:43:26.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:26.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj415 datafile 2024-09-18T14:43:26.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:26.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj416 datafile 2024-09-18T14:43:27.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:27.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj417 datafile 2024-09-18T14:43:27.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:27.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj418 datafile 2024-09-18T14:43:27.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:27.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj419 datafile 2024-09-18T14:43:27.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:27.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj420 datafile 2024-09-18T14:43:27.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:27.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj421 datafile 2024-09-18T14:43:27.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:27.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj422 datafile 2024-09-18T14:43:28.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:28.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj423 datafile 2024-09-18T14:43:28.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:28.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj424 datafile 2024-09-18T14:43:28.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:28.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj425 datafile 2024-09-18T14:43:28.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:28.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj426 datafile 2024-09-18T14:43:28.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:28.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj427 datafile 2024-09-18T14:43:28.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:28.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj428 datafile 2024-09-18T14:43:28.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:28.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj429 datafile 2024-09-18T14:43:29.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:29.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj430 datafile 2024-09-18T14:43:29.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:29.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj431 datafile 2024-09-18T14:43:29.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:29.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj432 datafile 2024-09-18T14:43:29.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:29.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj433 datafile 2024-09-18T14:43:29.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:29.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj434 datafile 2024-09-18T14:43:29.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:29.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj435 datafile 2024-09-18T14:43:30.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:30.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj436 datafile 2024-09-18T14:43:30.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:30.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj437 datafile 2024-09-18T14:43:30.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:30.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj438 datafile 2024-09-18T14:43:30.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:30.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj439 datafile 2024-09-18T14:43:30.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:30.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj440 datafile 2024-09-18T14:43:30.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:30.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj441 datafile 2024-09-18T14:43:31.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:31.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj442 datafile 2024-09-18T14:43:31.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:31.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj443 datafile 2024-09-18T14:43:31.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:31.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj444 datafile 2024-09-18T14:43:31.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:31.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj445 datafile 2024-09-18T14:43:31.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:31.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj446 datafile 2024-09-18T14:43:31.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:31.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj447 datafile 2024-09-18T14:43:31.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:31.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj448 datafile 2024-09-18T14:43:32.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:32.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj449 datafile 2024-09-18T14:43:32.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:32.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj450 datafile 2024-09-18T14:43:32.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:32.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj451 datafile 2024-09-18T14:43:32.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:32.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj452 datafile 2024-09-18T14:43:32.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:32.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj453 datafile 2024-09-18T14:43:32.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:32.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj454 datafile 2024-09-18T14:43:33.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:33.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj455 datafile 2024-09-18T14:43:33.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:33.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj456 datafile 2024-09-18T14:43:33.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:33.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj457 datafile 2024-09-18T14:43:33.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:33.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj458 datafile 2024-09-18T14:43:33.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:33.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj459 datafile 2024-09-18T14:43:33.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:33.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj460 datafile 2024-09-18T14:43:34.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:34.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj461 datafile 2024-09-18T14:43:34.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:34.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj462 datafile 2024-09-18T14:43:34.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:34.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj463 datafile 2024-09-18T14:43:34.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:34.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj464 datafile 2024-09-18T14:43:34.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:34.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj465 datafile 2024-09-18T14:43:34.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:34.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj466 datafile 2024-09-18T14:43:35.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:35.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj467 datafile 2024-09-18T14:43:35.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:35.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj468 datafile 2024-09-18T14:43:35.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:35.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj469 datafile 2024-09-18T14:43:35.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:35.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj470 datafile 2024-09-18T14:43:35.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:35.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj471 datafile 2024-09-18T14:43:35.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:35.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj472 datafile 2024-09-18T14:43:36.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:36.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj473 datafile 2024-09-18T14:43:36.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:36.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj474 datafile 2024-09-18T14:43:36.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:36.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj475 datafile 2024-09-18T14:43:36.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:36.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj476 datafile 2024-09-18T14:43:36.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:36.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj477 datafile 2024-09-18T14:43:36.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:36.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj478 datafile 2024-09-18T14:43:37.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:37.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj479 datafile 2024-09-18T14:43:37.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:37.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj480 datafile 2024-09-18T14:43:37.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:37.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj481 datafile 2024-09-18T14:43:37.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:37.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj482 datafile 2024-09-18T14:43:37.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:37.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj483 datafile 2024-09-18T14:43:37.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:37.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj484 datafile 2024-09-18T14:43:38.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:38.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj485 datafile 2024-09-18T14:43:38.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:38.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj486 datafile 2024-09-18T14:43:38.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:38.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj487 datafile 2024-09-18T14:43:38.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:38.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj488 datafile 2024-09-18T14:43:38.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:38.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj489 datafile 2024-09-18T14:43:38.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:38.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj490 datafile 2024-09-18T14:43:38.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:38.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj491 datafile 2024-09-18T14:43:39.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:39.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj492 datafile 2024-09-18T14:43:39.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:39.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj493 datafile 2024-09-18T14:43:39.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:39.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj494 datafile 2024-09-18T14:43:39.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:39.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj495 datafile 2024-09-18T14:43:39.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:39.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj496 datafile 2024-09-18T14:43:39.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:39.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj497 datafile 2024-09-18T14:43:40.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:40.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj498 datafile 2024-09-18T14:43:40.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:40.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj499 datafile 2024-09-18T14:43:40.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:40.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj500 datafile 2024-09-18T14:43:40.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:40.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj501 datafile 2024-09-18T14:43:40.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:40.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj502 datafile 2024-09-18T14:43:40.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:40.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj503 datafile 2024-09-18T14:43:41.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:41.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj504 datafile 2024-09-18T14:43:41.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:41.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj505 datafile 2024-09-18T14:43:41.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:41.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj506 datafile 2024-09-18T14:43:41.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:41.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj507 datafile 2024-09-18T14:43:41.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:41.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj508 datafile 2024-09-18T14:43:41.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:41.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj509 datafile 2024-09-18T14:43:42.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:42.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj510 datafile 2024-09-18T14:43:42.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:42.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj511 datafile 2024-09-18T14:43:42.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:42.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj512 datafile 2024-09-18T14:43:42.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:42.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj513 datafile 2024-09-18T14:43:42.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:42.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj514 datafile 2024-09-18T14:43:42.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:42.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj515 datafile 2024-09-18T14:43:42.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:42.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj516 datafile 2024-09-18T14:43:43.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:43.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj517 datafile 2024-09-18T14:43:43.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:43.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj518 datafile 2024-09-18T14:43:43.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:43.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj519 datafile 2024-09-18T14:43:43.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:43.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj520 datafile 2024-09-18T14:43:43.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:43.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj521 datafile 2024-09-18T14:43:43.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:43.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj522 datafile 2024-09-18T14:43:44.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:44.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj523 datafile 2024-09-18T14:43:44.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:44.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj524 datafile 2024-09-18T14:43:44.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:44.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj525 datafile 2024-09-18T14:43:44.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:44.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj526 datafile 2024-09-18T14:43:44.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:44.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj527 datafile 2024-09-18T14:43:44.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:44.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj528 datafile 2024-09-18T14:43:45.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:45.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj529 datafile 2024-09-18T14:43:45.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:45.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj530 datafile 2024-09-18T14:43:45.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:45.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj531 datafile 2024-09-18T14:43:45.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:45.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj532 datafile 2024-09-18T14:43:45.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:45.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj533 datafile 2024-09-18T14:43:45.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:45.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj534 datafile 2024-09-18T14:43:46.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:46.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj535 datafile 2024-09-18T14:43:46.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:46.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj536 datafile 2024-09-18T14:43:46.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:46.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj537 datafile 2024-09-18T14:43:46.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:46.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj538 datafile 2024-09-18T14:43:46.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:46.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj539 datafile 2024-09-18T14:43:46.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:46.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj540 datafile 2024-09-18T14:43:47.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:47.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj541 datafile 2024-09-18T14:43:47.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:47.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj542 datafile 2024-09-18T14:43:47.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:47.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj543 datafile 2024-09-18T14:43:47.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:47.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj544 datafile 2024-09-18T14:43:47.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:47.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj545 datafile 2024-09-18T14:43:47.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:47.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj546 datafile 2024-09-18T14:43:48.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:48.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj547 datafile 2024-09-18T14:43:48.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:48.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj548 datafile 2024-09-18T14:43:48.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:48.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj549 datafile 2024-09-18T14:43:48.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:48.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj550 datafile 2024-09-18T14:43:48.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:48.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj551 datafile 2024-09-18T14:43:48.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:48.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj552 datafile 2024-09-18T14:43:48.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:48.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj553 datafile 2024-09-18T14:43:49.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:49.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj554 datafile 2024-09-18T14:43:49.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:49.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj555 datafile 2024-09-18T14:43:49.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:49.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj556 datafile 2024-09-18T14:43:49.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:49.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj557 datafile 2024-09-18T14:43:49.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:49.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj558 datafile 2024-09-18T14:43:49.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:49.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj559 datafile 2024-09-18T14:43:50.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:50.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj560 datafile 2024-09-18T14:43:50.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:50.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj561 datafile 2024-09-18T14:43:50.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:50.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj562 datafile 2024-09-18T14:43:50.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:50.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj563 datafile 2024-09-18T14:43:51.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:51.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj564 datafile 2024-09-18T14:43:51.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:51.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj565 datafile 2024-09-18T14:43:51.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:51.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj566 datafile 2024-09-18T14:43:51.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:51.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj567 datafile 2024-09-18T14:43:51.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:51.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj568 datafile 2024-09-18T14:43:51.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:51.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj569 datafile 2024-09-18T14:43:52.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:52.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj570 datafile 2024-09-18T14:43:52.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:52.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj571 datafile 2024-09-18T14:43:52.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:52.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj572 datafile 2024-09-18T14:43:52.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:52.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj573 datafile 2024-09-18T14:43:52.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:52.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj574 datafile 2024-09-18T14:43:52.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:52.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj575 datafile 2024-09-18T14:43:52.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:52.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj576 datafile 2024-09-18T14:43:53.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:53.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj577 datafile 2024-09-18T14:43:53.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:53.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj578 datafile 2024-09-18T14:43:53.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:53.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj579 datafile 2024-09-18T14:43:53.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:53.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj580 datafile 2024-09-18T14:43:53.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:53.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj581 datafile 2024-09-18T14:43:54.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:54.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj582 datafile 2024-09-18T14:43:54.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:54.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj583 datafile 2024-09-18T14:43:54.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:54.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj584 datafile 2024-09-18T14:43:54.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:54.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj585 datafile 2024-09-18T14:43:54.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:54.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj586 datafile 2024-09-18T14:43:54.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:54.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj587 datafile 2024-09-18T14:43:55.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:55.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj588 datafile 2024-09-18T14:43:55.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:55.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj589 datafile 2024-09-18T14:43:55.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:55.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj590 datafile 2024-09-18T14:43:55.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:55.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj591 datafile 2024-09-18T14:43:55.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:55.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj592 datafile 2024-09-18T14:43:55.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:55.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj593 datafile 2024-09-18T14:43:56.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:56.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj594 datafile 2024-09-18T14:43:56.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:56.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj595 datafile 2024-09-18T14:43:56.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:56.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj596 datafile 2024-09-18T14:43:56.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:56.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj597 datafile 2024-09-18T14:43:56.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:56.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj598 datafile 2024-09-18T14:43:56.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:56.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj599 datafile 2024-09-18T14:43:57.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:57.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj600 datafile 2024-09-18T14:43:57.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:57.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj601 datafile 2024-09-18T14:43:57.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:57.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj602 datafile 2024-09-18T14:43:57.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:57.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj603 datafile 2024-09-18T14:43:57.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:57.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj604 datafile 2024-09-18T14:43:57.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:57.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj605 datafile 2024-09-18T14:43:58.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:58.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj606 datafile 2024-09-18T14:43:58.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:58.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj607 datafile 2024-09-18T14:43:58.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:58.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj608 datafile 2024-09-18T14:43:58.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:58.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj609 datafile 2024-09-18T14:43:58.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:58.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj610 datafile 2024-09-18T14:43:58.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:58.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj611 datafile 2024-09-18T14:43:59.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:59.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj612 datafile 2024-09-18T14:43:59.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:59.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj613 datafile 2024-09-18T14:43:59.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:59.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj614 datafile 2024-09-18T14:43:59.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:59.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj615 datafile 2024-09-18T14:43:59.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:59.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj616 datafile 2024-09-18T14:43:59.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:59.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj617 datafile 2024-09-18T14:43:59.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:43:59.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj618 datafile 2024-09-18T14:44:00.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:00.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj619 datafile 2024-09-18T14:44:00.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:00.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj620 datafile 2024-09-18T14:44:00.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:00.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj621 datafile 2024-09-18T14:44:00.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:00.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj622 datafile 2024-09-18T14:44:00.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:00.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj623 datafile 2024-09-18T14:44:00.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:00.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj624 datafile 2024-09-18T14:44:01.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:01.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj625 datafile 2024-09-18T14:44:01.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:01.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj626 datafile 2024-09-18T14:44:01.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:01.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj627 datafile 2024-09-18T14:44:01.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:01.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj628 datafile 2024-09-18T14:44:01.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:01.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj629 datafile 2024-09-18T14:44:02.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:02.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj630 datafile 2024-09-18T14:44:02.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:02.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj631 datafile 2024-09-18T14:44:02.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:02.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj632 datafile 2024-09-18T14:44:02.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:02.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj633 datafile 2024-09-18T14:44:02.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:02.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj634 datafile 2024-09-18T14:44:02.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:02.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj635 datafile 2024-09-18T14:44:02.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:02.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj636 datafile 2024-09-18T14:44:03.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:03.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj637 datafile 2024-09-18T14:44:03.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:03.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj638 datafile 2024-09-18T14:44:03.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:03.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj639 datafile 2024-09-18T14:44:03.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:03.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj640 datafile 2024-09-18T14:44:03.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:03.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj641 datafile 2024-09-18T14:44:03.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:03.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj642 datafile 2024-09-18T14:44:04.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:04.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj643 datafile 2024-09-18T14:44:04.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:04.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj644 datafile 2024-09-18T14:44:04.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:04.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj645 datafile 2024-09-18T14:44:04.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:04.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj646 datafile 2024-09-18T14:44:04.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:04.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj647 datafile 2024-09-18T14:44:04.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:04.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj648 datafile 2024-09-18T14:44:05.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:05.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj649 datafile 2024-09-18T14:44:05.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:05.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj650 datafile 2024-09-18T14:44:05.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:05.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj651 datafile 2024-09-18T14:44:05.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:05.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj652 datafile 2024-09-18T14:44:05.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:05.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj653 datafile 2024-09-18T14:44:05.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:05.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj654 datafile 2024-09-18T14:44:05.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:05.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj655 datafile 2024-09-18T14:44:06.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:06.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj656 datafile 2024-09-18T14:44:06.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:06.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj657 datafile 2024-09-18T14:44:06.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:06.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj658 datafile 2024-09-18T14:44:06.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:06.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj659 datafile 2024-09-18T14:44:06.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:06.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj660 datafile 2024-09-18T14:44:06.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:06.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj661 datafile 2024-09-18T14:44:07.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:07.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj662 datafile 2024-09-18T14:44:07.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:07.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj663 datafile 2024-09-18T14:44:07.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:07.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj664 datafile 2024-09-18T14:44:07.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:07.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj665 datafile 2024-09-18T14:44:07.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:07.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj666 datafile 2024-09-18T14:44:07.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:07.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj667 datafile 2024-09-18T14:44:08.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:08.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj668 datafile 2024-09-18T14:44:08.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:08.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj669 datafile 2024-09-18T14:44:08.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:08.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj670 datafile 2024-09-18T14:44:08.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:08.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj671 datafile 2024-09-18T14:44:08.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:08.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj672 datafile 2024-09-18T14:44:08.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:08.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj673 datafile 2024-09-18T14:44:09.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:09.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj674 datafile 2024-09-18T14:44:09.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:09.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj675 datafile 2024-09-18T14:44:09.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:09.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj676 datafile 2024-09-18T14:44:09.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:09.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj677 datafile 2024-09-18T14:44:09.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:09.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj678 datafile 2024-09-18T14:44:09.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:09.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj679 datafile 2024-09-18T14:44:10.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:10.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj680 datafile 2024-09-18T14:44:10.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:10.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj681 datafile 2024-09-18T14:44:10.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:10.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj682 datafile 2024-09-18T14:44:10.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:10.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj683 datafile 2024-09-18T14:44:10.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:10.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj684 datafile 2024-09-18T14:44:10.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:10.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj685 datafile 2024-09-18T14:44:11.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:11.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj686 datafile 2024-09-18T14:44:11.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:11.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj687 datafile 2024-09-18T14:44:11.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:11.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj688 datafile 2024-09-18T14:44:11.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:11.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj689 datafile 2024-09-18T14:44:11.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:11.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj690 datafile 2024-09-18T14:44:11.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:11.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj691 datafile 2024-09-18T14:44:12.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:12.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj692 datafile 2024-09-18T14:44:12.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:12.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj693 datafile 2024-09-18T14:44:12.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:12.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj694 datafile 2024-09-18T14:44:12.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:12.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj695 datafile 2024-09-18T14:44:12.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:12.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj696 datafile 2024-09-18T14:44:12.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:12.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj697 datafile 2024-09-18T14:44:13.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:13.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj698 datafile 2024-09-18T14:44:13.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:13.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj699 datafile 2024-09-18T14:44:13.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:13.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj700 datafile 2024-09-18T14:44:13.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:13.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj701 datafile 2024-09-18T14:44:13.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:13.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj702 datafile 2024-09-18T14:44:13.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:13.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj703 datafile 2024-09-18T14:44:14.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:14.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj704 datafile 2024-09-18T14:44:14.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:14.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj705 datafile 2024-09-18T14:44:14.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:14.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj706 datafile 2024-09-18T14:44:14.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:14.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj707 datafile 2024-09-18T14:44:14.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:14.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj708 datafile 2024-09-18T14:44:14.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:14.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj709 datafile 2024-09-18T14:44:15.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:15.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj710 datafile 2024-09-18T14:44:15.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:15.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj711 datafile 2024-09-18T14:44:15.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:15.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj712 datafile 2024-09-18T14:44:15.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:15.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj713 datafile 2024-09-18T14:44:15.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:15.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj714 datafile 2024-09-18T14:44:15.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:15.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj715 datafile 2024-09-18T14:44:16.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:16.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj716 datafile 2024-09-18T14:44:16.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:16.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj717 datafile 2024-09-18T14:44:16.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:16.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj718 datafile 2024-09-18T14:44:16.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:16.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj719 datafile 2024-09-18T14:44:16.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:16.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj720 datafile 2024-09-18T14:44:16.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:16.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj721 datafile 2024-09-18T14:44:16.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:16.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj722 datafile 2024-09-18T14:44:17.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:17.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj723 datafile 2024-09-18T14:44:17.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:17.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj724 datafile 2024-09-18T14:44:17.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:17.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj725 datafile 2024-09-18T14:44:17.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:17.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj726 datafile 2024-09-18T14:44:17.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:17.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj727 datafile 2024-09-18T14:44:18.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:18.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj728 datafile 2024-09-18T14:44:18.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:18.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj729 datafile 2024-09-18T14:44:18.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:18.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj730 datafile 2024-09-18T14:44:18.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:18.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj731 datafile 2024-09-18T14:44:18.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:18.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj732 datafile 2024-09-18T14:44:18.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:18.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj733 datafile 2024-09-18T14:44:19.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:19.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj734 datafile 2024-09-18T14:44:19.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:19.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj735 datafile 2024-09-18T14:44:19.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:19.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj736 datafile 2024-09-18T14:44:19.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:19.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj737 datafile 2024-09-18T14:44:19.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:19.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj738 datafile 2024-09-18T14:44:19.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:19.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj739 datafile 2024-09-18T14:44:19.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:19.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj740 datafile 2024-09-18T14:44:20.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:20.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj741 datafile 2024-09-18T14:44:20.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:20.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj742 datafile 2024-09-18T14:44:20.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:20.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj743 datafile 2024-09-18T14:44:20.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:20.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj744 datafile 2024-09-18T14:44:20.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:20.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj745 datafile 2024-09-18T14:44:20.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:20.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj746 datafile 2024-09-18T14:44:21.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:21.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj747 datafile 2024-09-18T14:44:21.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:21.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj748 datafile 2024-09-18T14:44:21.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:21.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj749 datafile 2024-09-18T14:44:22.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:22.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj750 datafile 2024-09-18T14:44:22.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:22.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj751 datafile 2024-09-18T14:44:22.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:22.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj752 datafile 2024-09-18T14:44:22.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:22.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj753 datafile 2024-09-18T14:44:22.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:22.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj754 datafile 2024-09-18T14:44:22.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:22.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj755 datafile 2024-09-18T14:44:22.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:22.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj756 datafile 2024-09-18T14:44:23.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:23.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj757 datafile 2024-09-18T14:44:23.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:23.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj758 datafile 2024-09-18T14:44:23.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:23.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj759 datafile 2024-09-18T14:44:23.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:23.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj760 datafile 2024-09-18T14:44:23.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:23.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj761 datafile 2024-09-18T14:44:23.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:23.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj762 datafile 2024-09-18T14:44:24.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:24.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj763 datafile 2024-09-18T14:44:24.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:24.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj764 datafile 2024-09-18T14:44:24.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:24.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj765 datafile 2024-09-18T14:44:24.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:24.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj766 datafile 2024-09-18T14:44:24.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:24.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj767 datafile 2024-09-18T14:44:24.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:24.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj768 datafile 2024-09-18T14:44:25.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:25.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj769 datafile 2024-09-18T14:44:25.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:25.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj770 datafile 2024-09-18T14:44:25.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:25.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj771 datafile 2024-09-18T14:44:25.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:25.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj772 datafile 2024-09-18T14:44:25.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:25.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj773 datafile 2024-09-18T14:44:25.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:25.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj774 datafile 2024-09-18T14:44:25.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:25.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj775 datafile 2024-09-18T14:44:26.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:26.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj776 datafile 2024-09-18T14:44:26.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:26.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj777 datafile 2024-09-18T14:44:26.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:26.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj778 datafile 2024-09-18T14:44:26.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:26.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj779 datafile 2024-09-18T14:44:26.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:26.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj780 datafile 2024-09-18T14:44:27.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:27.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj781 datafile 2024-09-18T14:44:27.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:27.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj782 datafile 2024-09-18T14:44:27.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:27.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj783 datafile 2024-09-18T14:44:27.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:27.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj784 datafile 2024-09-18T14:44:27.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:27.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj785 datafile 2024-09-18T14:44:27.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:27.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj786 datafile 2024-09-18T14:44:27.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:27.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj787 datafile 2024-09-18T14:44:28.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:28.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj788 datafile 2024-09-18T14:44:28.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:28.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj789 datafile 2024-09-18T14:44:28.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:28.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj790 datafile 2024-09-18T14:44:28.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:28.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj791 datafile 2024-09-18T14:44:28.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:28.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj792 datafile 2024-09-18T14:44:28.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:28.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj793 datafile 2024-09-18T14:44:29.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:29.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj794 datafile 2024-09-18T14:44:29.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:29.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj795 datafile 2024-09-18T14:44:29.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:29.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj796 datafile 2024-09-18T14:44:29.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:29.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj797 datafile 2024-09-18T14:44:29.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:29.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj798 datafile 2024-09-18T14:44:29.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:29.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj799 datafile 2024-09-18T14:44:30.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:30.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj800 datafile 2024-09-18T14:44:30.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:30.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj801 datafile 2024-09-18T14:44:30.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:30.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj802 datafile 2024-09-18T14:44:30.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:30.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj803 datafile 2024-09-18T14:44:30.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:30.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj804 datafile 2024-09-18T14:44:30.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:30.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj805 datafile 2024-09-18T14:44:30.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:30.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj806 datafile 2024-09-18T14:44:31.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:31.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj807 datafile 2024-09-18T14:44:31.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:31.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj808 datafile 2024-09-18T14:44:31.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:31.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj809 datafile 2024-09-18T14:44:31.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:31.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj810 datafile 2024-09-18T14:44:31.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:31.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj811 datafile 2024-09-18T14:44:31.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:31.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj812 datafile 2024-09-18T14:44:32.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:32.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj813 datafile 2024-09-18T14:44:32.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:32.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj814 datafile 2024-09-18T14:44:32.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:32.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj815 datafile 2024-09-18T14:44:32.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:32.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj816 datafile 2024-09-18T14:44:32.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:32.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj817 datafile 2024-09-18T14:44:32.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:32.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj818 datafile 2024-09-18T14:44:33.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:33.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj819 datafile 2024-09-18T14:44:33.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:33.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj820 datafile 2024-09-18T14:44:33.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:33.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj821 datafile 2024-09-18T14:44:33.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:33.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj822 datafile 2024-09-18T14:44:33.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:33.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj823 datafile 2024-09-18T14:44:33.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:33.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj824 datafile 2024-09-18T14:44:33.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:33.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj825 datafile 2024-09-18T14:44:34.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:34.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj826 datafile 2024-09-18T14:44:34.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:34.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj827 datafile 2024-09-18T14:44:34.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:34.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj828 datafile 2024-09-18T14:44:34.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:34.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj829 datafile 2024-09-18T14:44:34.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:34.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj830 datafile 2024-09-18T14:44:34.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:34.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj831 datafile 2024-09-18T14:44:35.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:35.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj832 datafile 2024-09-18T14:44:35.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:35.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj833 datafile 2024-09-18T14:44:35.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:35.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj834 datafile 2024-09-18T14:44:35.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:35.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj835 datafile 2024-09-18T14:44:35.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:35.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj836 datafile 2024-09-18T14:44:36.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:36.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj837 datafile 2024-09-18T14:44:36.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:36.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj838 datafile 2024-09-18T14:44:36.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:36.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj839 datafile 2024-09-18T14:44:36.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:36.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj840 datafile 2024-09-18T14:44:36.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:36.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj841 datafile 2024-09-18T14:44:36.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:36.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj842 datafile 2024-09-18T14:44:36.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:36.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj843 datafile 2024-09-18T14:44:37.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:37.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj844 datafile 2024-09-18T14:44:37.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:37.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj845 datafile 2024-09-18T14:44:37.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:37.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj846 datafile 2024-09-18T14:44:37.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:37.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj847 datafile 2024-09-18T14:44:37.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:37.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj848 datafile 2024-09-18T14:44:38.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:38.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj849 datafile 2024-09-18T14:44:38.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:38.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj850 datafile 2024-09-18T14:44:38.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:38.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj851 datafile 2024-09-18T14:44:38.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:38.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj852 datafile 2024-09-18T14:44:38.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:38.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj853 datafile 2024-09-18T14:44:38.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:38.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj854 datafile 2024-09-18T14:44:38.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:38.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj855 datafile 2024-09-18T14:44:39.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:39.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj856 datafile 2024-09-18T14:44:39.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:39.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj857 datafile 2024-09-18T14:44:39.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:39.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj858 datafile 2024-09-18T14:44:39.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:39.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj859 datafile 2024-09-18T14:44:39.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:39.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj860 datafile 2024-09-18T14:44:39.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:39.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj861 datafile 2024-09-18T14:44:40.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:40.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj862 datafile 2024-09-18T14:44:40.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:40.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj863 datafile 2024-09-18T14:44:40.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:40.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj864 datafile 2024-09-18T14:44:40.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:40.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj865 datafile 2024-09-18T14:44:40.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:40.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj866 datafile 2024-09-18T14:44:40.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:40.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj867 datafile 2024-09-18T14:44:41.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:41.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj868 datafile 2024-09-18T14:44:41.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:41.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj869 datafile 2024-09-18T14:44:41.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:41.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj870 datafile 2024-09-18T14:44:41.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:41.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj871 datafile 2024-09-18T14:44:41.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:41.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj872 datafile 2024-09-18T14:44:41.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:41.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj873 datafile 2024-09-18T14:44:42.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:42.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj874 datafile 2024-09-18T14:44:42.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:42.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj875 datafile 2024-09-18T14:44:42.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:42.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj876 datafile 2024-09-18T14:44:42.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:42.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj877 datafile 2024-09-18T14:44:42.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:42.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj878 datafile 2024-09-18T14:44:42.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:42.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj879 datafile 2024-09-18T14:44:42.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:42.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj880 datafile 2024-09-18T14:44:43.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:43.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj881 datafile 2024-09-18T14:44:43.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:43.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj882 datafile 2024-09-18T14:44:43.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:43.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj883 datafile 2024-09-18T14:44:43.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:43.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj884 datafile 2024-09-18T14:44:43.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:43.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj885 datafile 2024-09-18T14:44:43.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:43.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj886 datafile 2024-09-18T14:44:44.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:44.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj887 datafile 2024-09-18T14:44:44.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:44.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj888 datafile 2024-09-18T14:44:44.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:44.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj889 datafile 2024-09-18T14:44:44.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:44.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj890 datafile 2024-09-18T14:44:44.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:44.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj891 datafile 2024-09-18T14:44:44.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:44.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj892 datafile 2024-09-18T14:44:44.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:44.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj893 datafile 2024-09-18T14:44:45.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:45.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj894 datafile 2024-09-18T14:44:45.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:45.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj895 datafile 2024-09-18T14:44:45.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:45.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj896 datafile 2024-09-18T14:44:45.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:45.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj897 datafile 2024-09-18T14:44:45.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:45.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj898 datafile 2024-09-18T14:44:46.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:46.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj899 datafile 2024-09-18T14:44:46.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:46.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj900 datafile 2024-09-18T14:44:46.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:46.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj901 datafile 2024-09-18T14:44:46.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:46.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj902 datafile 2024-09-18T14:44:46.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:46.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj903 datafile 2024-09-18T14:44:46.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:46.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj904 datafile 2024-09-18T14:44:46.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:46.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj905 datafile 2024-09-18T14:44:47.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:47.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj906 datafile 2024-09-18T14:44:47.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:47.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj907 datafile 2024-09-18T14:44:47.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:47.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj908 datafile 2024-09-18T14:44:47.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:47.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj909 datafile 2024-09-18T14:44:47.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:47.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj910 datafile 2024-09-18T14:44:47.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:47.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj911 datafile 2024-09-18T14:44:48.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:48.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj912 datafile 2024-09-18T14:44:48.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:48.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj913 datafile 2024-09-18T14:44:48.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:48.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj914 datafile 2024-09-18T14:44:48.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:48.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj915 datafile 2024-09-18T14:44:48.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:48.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj916 datafile 2024-09-18T14:44:48.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:48.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj917 datafile 2024-09-18T14:44:48.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:48.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj918 datafile 2024-09-18T14:44:49.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:49.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj919 datafile 2024-09-18T14:44:49.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:49.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj920 datafile 2024-09-18T14:44:49.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:49.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj921 datafile 2024-09-18T14:44:49.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:49.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj922 datafile 2024-09-18T14:44:49.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:49.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj923 datafile 2024-09-18T14:44:50.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:50.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj924 datafile 2024-09-18T14:44:50.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:50.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj925 datafile 2024-09-18T14:44:50.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:50.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj926 datafile 2024-09-18T14:44:50.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:50.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj927 datafile 2024-09-18T14:44:50.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:50.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj928 datafile 2024-09-18T14:44:50.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:50.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj929 datafile 2024-09-18T14:44:50.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:50.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj930 datafile 2024-09-18T14:44:51.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:51.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj931 datafile 2024-09-18T14:44:51.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:51.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj932 datafile 2024-09-18T14:44:51.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:51.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj933 datafile 2024-09-18T14:44:51.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:51.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj934 datafile 2024-09-18T14:44:51.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:51.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj935 datafile 2024-09-18T14:44:51.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:51.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj936 datafile 2024-09-18T14:44:52.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:52.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj937 datafile 2024-09-18T14:44:52.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:52.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj938 datafile 2024-09-18T14:44:52.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:52.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj939 datafile 2024-09-18T14:44:52.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:52.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj940 datafile 2024-09-18T14:44:52.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:52.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj941 datafile 2024-09-18T14:44:53.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:53.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj942 datafile 2024-09-18T14:44:53.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:53.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj943 datafile 2024-09-18T14:44:53.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:53.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj944 datafile 2024-09-18T14:44:53.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:53.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj945 datafile 2024-09-18T14:44:53.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:53.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj946 datafile 2024-09-18T14:44:53.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:53.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj947 datafile 2024-09-18T14:44:54.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:54.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj948 datafile 2024-09-18T14:44:54.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:54.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj949 datafile 2024-09-18T14:44:54.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:54.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj950 datafile 2024-09-18T14:44:54.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:54.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj951 datafile 2024-09-18T14:44:54.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:54.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj952 datafile 2024-09-18T14:44:54.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:54.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj953 datafile 2024-09-18T14:44:55.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:55.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj954 datafile 2024-09-18T14:44:55.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:55.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj955 datafile 2024-09-18T14:44:55.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:55.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj956 datafile 2024-09-18T14:44:55.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:55.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj957 datafile 2024-09-18T14:44:55.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:55.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj958 datafile 2024-09-18T14:44:55.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:55.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj959 datafile 2024-09-18T14:44:56.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:56.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj960 datafile 2024-09-18T14:44:56.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:56.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj961 datafile 2024-09-18T14:44:56.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:56.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj962 datafile 2024-09-18T14:44:56.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:56.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj963 datafile 2024-09-18T14:44:56.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:56.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj964 datafile 2024-09-18T14:44:56.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:56.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj965 datafile 2024-09-18T14:44:57.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:57.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj966 datafile 2024-09-18T14:44:57.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:57.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj967 datafile 2024-09-18T14:44:57.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:57.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj968 datafile 2024-09-18T14:44:57.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:57.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj969 datafile 2024-09-18T14:44:57.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:57.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj970 datafile 2024-09-18T14:44:57.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:57.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj971 datafile 2024-09-18T14:44:58.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:58.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj972 datafile 2024-09-18T14:44:58.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:58.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj973 datafile 2024-09-18T14:44:58.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:58.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj974 datafile 2024-09-18T14:44:58.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:58.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj975 datafile 2024-09-18T14:44:58.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:58.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj976 datafile 2024-09-18T14:44:58.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:58.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj977 datafile 2024-09-18T14:44:59.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:59.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj978 datafile 2024-09-18T14:44:59.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:59.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj979 datafile 2024-09-18T14:44:59.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:59.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj980 datafile 2024-09-18T14:44:59.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:59.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj981 datafile 2024-09-18T14:44:59.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:59.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj982 datafile 2024-09-18T14:44:59.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:44:59.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj983 datafile 2024-09-18T14:45:00.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:45:00.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj984 datafile 2024-09-18T14:45:00.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:45:00.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj985 datafile 2024-09-18T14:45:00.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:45:00.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj986 datafile 2024-09-18T14:45:00.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:45:00.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj987 datafile 2024-09-18T14:45:00.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:45:00.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj988 datafile 2024-09-18T14:45:00.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:45:00.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj989 datafile 2024-09-18T14:45:01.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:45:01.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj990 datafile 2024-09-18T14:45:01.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:45:01.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj991 datafile 2024-09-18T14:45:01.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:45:01.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj992 datafile 2024-09-18T14:45:01.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:45:01.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj993 datafile 2024-09-18T14:45:01.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:45:01.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj994 datafile 2024-09-18T14:45:01.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:45:01.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj995 datafile 2024-09-18T14:45:01.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:45:01.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj996 datafile 2024-09-18T14:45:02.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:45:02.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj997 datafile 2024-09-18T14:45:02.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:45:02.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj998 datafile 2024-09-18T14:45:02.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:45:02.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj999 datafile 2024-09-18T14:45:02.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2024-09-18T14:45:02.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:78: TEST_recover_unexpected: rados -p test3 put obj1000 datafile 2024-09-18T14:45:02.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:81: TEST_recover_unexpected: rm datafile 2024-09-18T14:45:02.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:83: TEST_recover_unexpected: ceph osd set noscrub 2024-09-18T14:45:03.270 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T14:45:03.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: ceph osd set nodeep-scrub 2024-09-18T14:45:03.923 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T14:45:03.938 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json-pretty 2024-09-18T14:45:03.938 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[].pgid' 2024-09-18T14:45:04.306 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:04.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:04.318 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:04.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.9") | .acting_primary' 2024-09-18T14:45:04.629 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:04.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=4 2024-09-18T14:45:04.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="3.9"' 2024-09-18T14:45:04.640 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=3.9 2024-09-18T14:45:04.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 3.9 scrub 2024-09-18T14:45:04.778 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:04.778 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:04.778 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:24.774712+0000" 2024-09-18T14:45:04.778 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:04.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:04.791 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:04.791 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.b") | .acting_primary' 2024-09-18T14:45:05.111 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:05.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=1 2024-09-18T14:45:05.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="1.b"' 2024-09-18T14:45:05.129 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=1.b 2024-09-18T14:45:05.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 1.b scrub 2024-09-18T14:45:05.263 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:05.263 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:05.263 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:25.259507+0000" 2024-09-18T14:45:05.263 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:05.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:05.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:05.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.8") | .acting_primary' 2024-09-18T14:45:05.583 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:05.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=0 2024-09-18T14:45:05.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="2.8"' 2024-09-18T14:45:05.595 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=2.8 2024-09-18T14:45:05.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 2.8 scrub 2024-09-18T14:45:05.735 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:05.735 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:05.735 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:25.731609+0000" 2024-09-18T14:45:05.735 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:05.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:05.747 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:05.747 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.8") | .acting_primary' 2024-09-18T14:45:06.057 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:06.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=4 2024-09-18T14:45:06.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="3.8"' 2024-09-18T14:45:06.068 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=3.8 2024-09-18T14:45:06.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 3.8 scrub 2024-09-18T14:45:06.204 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:06.205 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:06.205 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:26.201260+0000" 2024-09-18T14:45:06.205 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:06.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:06.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:06.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.a") | .acting_primary' 2024-09-18T14:45:06.530 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:06.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=1 2024-09-18T14:45:06.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="1.a"' 2024-09-18T14:45:06.542 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=1.a 2024-09-18T14:45:06.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 1.a scrub 2024-09-18T14:45:06.698 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:06.698 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:06.698 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:26.694407+0000" 2024-09-18T14:45:06.698 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:06.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:06.714 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:06.714 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.9") | .acting_primary' 2024-09-18T14:45:07.024 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:07.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=4 2024-09-18T14:45:07.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="2.9"' 2024-09-18T14:45:07.037 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=2.9 2024-09-18T14:45:07.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 2.9 scrub 2024-09-18T14:45:07.170 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:07.170 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:07.170 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:27.166635+0000" 2024-09-18T14:45:07.170 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:07.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:07.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:07.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.b") | .acting_primary' 2024-09-18T14:45:07.528 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:07.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=3 2024-09-18T14:45:07.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="3.b"' 2024-09-18T14:45:07.540 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=3.b 2024-09-18T14:45:07.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 3.b scrub 2024-09-18T14:45:07.679 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:07.679 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:07.679 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:27.676050+0000" 2024-09-18T14:45:07.679 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:07.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:07.695 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:07.696 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.9") | .acting_primary' 2024-09-18T14:45:08.005 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:08.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=1 2024-09-18T14:45:08.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="1.9"' 2024-09-18T14:45:08.015 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=1.9 2024-09-18T14:45:08.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 1.9 scrub 2024-09-18T14:45:08.155 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:08.155 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:08.155 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:28.151548+0000" 2024-09-18T14:45:08.155 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:08.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:08.167 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:08.167 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.a") | .acting_primary' 2024-09-18T14:45:08.518 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:08.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=1 2024-09-18T14:45:08.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="2.a"' 2024-09-18T14:45:08.528 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=2.a 2024-09-18T14:45:08.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 2.a scrub 2024-09-18T14:45:08.668 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:08.669 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:08.669 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:28.665056+0000" 2024-09-18T14:45:08.669 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:08.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:08.681 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.a") | .acting_primary' 2024-09-18T14:45:08.681 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:08.995 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:09.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=3 2024-09-18T14:45:09.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="3.a"' 2024-09-18T14:45:09.008 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=3.a 2024-09-18T14:45:09.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 3.a scrub 2024-09-18T14:45:09.148 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:09.148 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:09.148 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:29.144417+0000" 2024-09-18T14:45:09.148 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:09.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:09.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:09.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.8") | .acting_primary' 2024-09-18T14:45:09.473 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:09.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=5 2024-09-18T14:45:09.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="1.8"' 2024-09-18T14:45:09.485 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=1.8 2024-09-18T14:45:09.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 1.8 scrub 2024-09-18T14:45:09.619 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:09.619 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:09.619 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:29.615673+0000" 2024-09-18T14:45:09.619 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:09.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:09.631 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:09.631 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.b") | .acting_primary' 2024-09-18T14:45:09.949 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:09.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=3 2024-09-18T14:45:09.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="2.b"' 2024-09-18T14:45:09.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=2.b 2024-09-18T14:45:09.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 2.b scrub 2024-09-18T14:45:10.101 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:10.101 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:10.102 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:30.097882+0000" 2024-09-18T14:45:10.102 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:10.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:10.114 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:10.114 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.5") | .acting_primary' 2024-09-18T14:45:10.433 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:10.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=5 2024-09-18T14:45:10.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="3.5"' 2024-09-18T14:45:10.446 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=3.5 2024-09-18T14:45:10.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 3.5 scrub 2024-09-18T14:45:10.584 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:10.584 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:10.584 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:30.580906+0000" 2024-09-18T14:45:10.584 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:10.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:10.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:10.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.7") | .acting_primary' 2024-09-18T14:45:10.912 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:10.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=1 2024-09-18T14:45:10.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="1.7"' 2024-09-18T14:45:10.924 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=1.7 2024-09-18T14:45:10.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 1.7 scrub 2024-09-18T14:45:11.065 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:11.065 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:11.065 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:31.061428+0000" 2024-09-18T14:45:11.065 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:11.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:11.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:11.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.4") | .acting_primary' 2024-09-18T14:45:11.392 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:11.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=1 2024-09-18T14:45:11.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="2.4"' 2024-09-18T14:45:11.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=2.4 2024-09-18T14:45:11.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 2.4 scrub 2024-09-18T14:45:11.539 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:11.539 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:11.539 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:31.535083+0000" 2024-09-18T14:45:11.539 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:11.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:11.550 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.4") | .acting_primary' 2024-09-18T14:45:11.551 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:11.868 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:11.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=1 2024-09-18T14:45:11.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="3.4"' 2024-09-18T14:45:11.880 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=3.4 2024-09-18T14:45:11.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 3.4 scrub 2024-09-18T14:45:12.023 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:12.023 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:12.023 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:32.019342+0000" 2024-09-18T14:45:12.023 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:12.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:12.035 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:12.035 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.6") | .acting_primary' 2024-09-18T14:45:12.350 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:12.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=3 2024-09-18T14:45:12.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="1.6"' 2024-09-18T14:45:12.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=1.6 2024-09-18T14:45:12.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 1.6 scrub 2024-09-18T14:45:12.502 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:12.502 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:12.502 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:32.498437+0000" 2024-09-18T14:45:12.502 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:12.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:12.518 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:12.518 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.5") | .acting_primary' 2024-09-18T14:45:12.839 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:12.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=3 2024-09-18T14:45:12.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="2.5"' 2024-09-18T14:45:12.851 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=2.5 2024-09-18T14:45:12.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 2.5 scrub 2024-09-18T14:45:12.993 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:12.994 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:12.994 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:32.990299+0000" 2024-09-18T14:45:12.994 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:13.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:13.006 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:13.006 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.7") | .acting_primary' 2024-09-18T14:45:13.316 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:13.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=4 2024-09-18T14:45:13.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="3.7"' 2024-09-18T14:45:13.328 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=3.7 2024-09-18T14:45:13.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 3.7 scrub 2024-09-18T14:45:13.465 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:13.465 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:13.465 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:33.461488+0000" 2024-09-18T14:45:13.465 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:13.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:13.476 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:13.476 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.5") | .acting_primary' 2024-09-18T14:45:13.792 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:13.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=4 2024-09-18T14:45:13.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="1.5"' 2024-09-18T14:45:13.803 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=1.5 2024-09-18T14:45:13.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 1.5 scrub 2024-09-18T14:45:13.934 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:13.934 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:13.934 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:33.930364+0000" 2024-09-18T14:45:13.934 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:13.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:13.946 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:13.946 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.6") | .acting_primary' 2024-09-18T14:45:14.258 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:14.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=1 2024-09-18T14:45:14.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="2.6"' 2024-09-18T14:45:14.270 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=2.6 2024-09-18T14:45:14.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 2.6 scrub 2024-09-18T14:45:14.434 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:14.434 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:14.434 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:34.430687+0000" 2024-09-18T14:45:14.434 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:14.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:14.447 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:14.447 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.6") | .acting_primary' 2024-09-18T14:45:14.767 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:14.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=0 2024-09-18T14:45:14.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="3.6"' 2024-09-18T14:45:14.780 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=3.6 2024-09-18T14:45:14.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 3.6 scrub 2024-09-18T14:45:14.913 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:14.913 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:14.913 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:34.909707+0000" 2024-09-18T14:45:14.914 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:14.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:14.925 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:14.926 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.4") | .acting_primary' 2024-09-18T14:45:15.237 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:15.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=3 2024-09-18T14:45:15.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="1.4"' 2024-09-18T14:45:15.247 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=1.4 2024-09-18T14:45:15.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 1.4 scrub 2024-09-18T14:45:15.411 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:15.411 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:15.411 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:35.407291+0000" 2024-09-18T14:45:15.411 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:15.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:15.423 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:15.424 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.7") | .acting_primary' 2024-09-18T14:45:15.730 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:15.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=3 2024-09-18T14:45:15.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="2.7"' 2024-09-18T14:45:15.742 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=2.7 2024-09-18T14:45:15.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 2.7 scrub 2024-09-18T14:45:15.883 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:15.883 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:15.883 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:35.879954+0000" 2024-09-18T14:45:15.884 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:15.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:15.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:15.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.1") | .acting_primary' 2024-09-18T14:45:16.204 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:16.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=4 2024-09-18T14:45:16.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="3.1"' 2024-09-18T14:45:16.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=3.1 2024-09-18T14:45:16.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 3.1 scrub 2024-09-18T14:45:16.445 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:16.445 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:16.445 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:36.441821+0000" 2024-09-18T14:45:16.445 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:16.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:16.458 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:16.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.3") | .acting_primary' 2024-09-18T14:45:16.780 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:16.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=4 2024-09-18T14:45:16.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="1.3"' 2024-09-18T14:45:16.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=1.3 2024-09-18T14:45:16.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 1.3 scrub 2024-09-18T14:45:16.932 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:16.932 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:16.933 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:36.928924+0000" 2024-09-18T14:45:16.933 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:16.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:16.945 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:16.945 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.0") | .acting_primary' 2024-09-18T14:45:17.269 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:17.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=3 2024-09-18T14:45:17.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="2.0"' 2024-09-18T14:45:17.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=2.0 2024-09-18T14:45:17.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 2.0 scrub 2024-09-18T14:45:17.420 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:17.421 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:17.421 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:37.417204+0000" 2024-09-18T14:45:17.421 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:17.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:17.433 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:17.433 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.0") | .acting_primary' 2024-09-18T14:45:17.743 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:17.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=1 2024-09-18T14:45:17.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="3.0"' 2024-09-18T14:45:17.755 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=3.0 2024-09-18T14:45:17.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 3.0 scrub 2024-09-18T14:45:17.898 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:17.898 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:17.899 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:37.894982+0000" 2024-09-18T14:45:17.899 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:17.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:17.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:17.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.2") | .acting_primary' 2024-09-18T14:45:18.223 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:18.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=0 2024-09-18T14:45:18.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="1.2"' 2024-09-18T14:45:18.238 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=1.2 2024-09-18T14:45:18.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 1.2 scrub 2024-09-18T14:45:18.440 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:18.440 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:18.441 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:38.436840+0000" 2024-09-18T14:45:18.441 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:18.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:18.454 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:18.454 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.1") | .acting_primary' 2024-09-18T14:45:18.767 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:18.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=2 2024-09-18T14:45:18.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="2.1"' 2024-09-18T14:45:18.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=2.1 2024-09-18T14:45:18.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 2.1 scrub 2024-09-18T14:45:18.917 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:18.917 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:18.917 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:38.913415+0000" 2024-09-18T14:45:18.917 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:18.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:18.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:18.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.e") | .acting_primary' 2024-09-18T14:45:19.242 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:19.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=2 2024-09-18T14:45:19.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="3.e"' 2024-09-18T14:45:19.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=3.e 2024-09-18T14:45:19.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 3.e scrub 2024-09-18T14:45:19.405 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:19.405 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:19.405 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:39.401311+0000" 2024-09-18T14:45:19.405 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:19.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:19.417 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:19.417 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.f") | .acting_primary' 2024-09-18T14:45:19.731 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:19.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=0 2024-09-18T14:45:19.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="2.f"' 2024-09-18T14:45:19.744 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=2.f 2024-09-18T14:45:19.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 2.f scrub 2024-09-18T14:45:19.882 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:19.882 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:19.883 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:39.878926+0000" 2024-09-18T14:45:19.883 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:19.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:19.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:19.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.c") | .acting_primary' 2024-09-18T14:45:20.204 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:20.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=3 2024-09-18T14:45:20.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="1.c"' 2024-09-18T14:45:20.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=1.c 2024-09-18T14:45:20.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 1.c scrub 2024-09-18T14:45:20.351 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:20.351 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:20.351 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:40.347430+0000" 2024-09-18T14:45:20.351 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:20.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:20.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:20.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.3") | .acting_primary' 2024-09-18T14:45:20.680 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:20.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=4 2024-09-18T14:45:20.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="3.3"' 2024-09-18T14:45:20.696 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=3.3 2024-09-18T14:45:20.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 3.3 scrub 2024-09-18T14:45:20.834 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:20.834 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:20.834 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:40.831017+0000" 2024-09-18T14:45:20.834 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:20.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:20.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:20.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.1") | .acting_primary' 2024-09-18T14:45:21.161 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:21.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=5 2024-09-18T14:45:21.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="1.1"' 2024-09-18T14:45:21.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=1.1 2024-09-18T14:45:21.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 1.1 scrub 2024-09-18T14:45:21.349 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:21.349 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:21.349 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:41.345867+0000" 2024-09-18T14:45:21.349 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:21.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:21.362 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:21.362 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.2") | .acting_primary' 2024-09-18T14:45:21.675 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:21.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=5 2024-09-18T14:45:21.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="2.2"' 2024-09-18T14:45:21.687 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=2.2 2024-09-18T14:45:21.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 2.2 scrub 2024-09-18T14:45:21.817 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:21.817 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:21.817 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:41.813881+0000" 2024-09-18T14:45:21.817 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:21.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:21.830 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:21.830 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.2") | .acting_primary' 2024-09-18T14:45:22.145 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:22.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=3 2024-09-18T14:45:22.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="3.2"' 2024-09-18T14:45:22.159 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=3.2 2024-09-18T14:45:22.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 3.2 scrub 2024-09-18T14:45:22.303 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:22.303 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:22.303 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:42.300039+0000" 2024-09-18T14:45:22.303 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:22.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:22.316 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:22.316 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.0") | .acting_primary' 2024-09-18T14:45:22.647 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:22.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=1 2024-09-18T14:45:22.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="1.0"' 2024-09-18T14:45:22.660 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=1.0 2024-09-18T14:45:22.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 1.0 scrub 2024-09-18T14:45:22.789 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:22.789 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:22.789 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:42.785839+0000" 2024-09-18T14:45:22.789 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:22.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:22.802 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:22.802 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.3") | .acting_primary' 2024-09-18T14:45:23.115 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:23.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=5 2024-09-18T14:45:23.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="2.3"' 2024-09-18T14:45:23.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=2.3 2024-09-18T14:45:23.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 2.3 scrub 2024-09-18T14:45:23.265 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:23.265 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:23.265 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:43.262049+0000" 2024-09-18T14:45:23.266 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:23.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:23.277 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:23.277 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.f") | .acting_primary' 2024-09-18T14:45:23.589 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:23.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=0 2024-09-18T14:45:23.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="3.f"' 2024-09-18T14:45:23.600 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=3.f 2024-09-18T14:45:23.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 3.f scrub 2024-09-18T14:45:23.743 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:23.743 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:23.743 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:43.739831+0000" 2024-09-18T14:45:23.743 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:23.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:23.755 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:23.756 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.e") | .acting_primary' 2024-09-18T14:45:24.057 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:24.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=2 2024-09-18T14:45:24.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="2.e"' 2024-09-18T14:45:24.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=2.e 2024-09-18T14:45:24.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 2.e scrub 2024-09-18T14:45:24.212 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:24.213 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:24.213 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:44.209216+0000" 2024-09-18T14:45:24.213 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:24.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:24.225 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:24.225 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.d") | .acting_primary' 2024-09-18T14:45:24.584 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:24.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=2 2024-09-18T14:45:24.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="1.d"' 2024-09-18T14:45:24.596 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=1.d 2024-09-18T14:45:24.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 1.d scrub 2024-09-18T14:45:24.729 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:24.729 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:24.729 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:44.725794+0000" 2024-09-18T14:45:24.729 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:24.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:24.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:24.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.c") | .acting_primary' 2024-09-18T14:45:25.053 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:25.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=5 2024-09-18T14:45:25.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="3.c"' 2024-09-18T14:45:25.068 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=3.c 2024-09-18T14:45:25.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 3.c scrub 2024-09-18T14:45:25.212 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:25.220 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:25.220 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:45.208366+0000" 2024-09-18T14:45:25.220 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:25.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:25.224 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:25.224 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.d") | .acting_primary' 2024-09-18T14:45:25.581 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:25.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=5 2024-09-18T14:45:25.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="2.d"' 2024-09-18T14:45:25.594 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=2.d 2024-09-18T14:45:25.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 2.d scrub 2024-09-18T14:45:25.734 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:25.734 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:25.734 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:45.730754+0000" 2024-09-18T14:45:25.734 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:25.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:25.747 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:25.747 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.e") | .acting_primary' 2024-09-18T14:45:26.064 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:26.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=1 2024-09-18T14:45:26.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="1.e"' 2024-09-18T14:45:26.077 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=1.e 2024-09-18T14:45:26.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 1.e scrub 2024-09-18T14:45:26.214 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:26.214 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:26.214 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:46.210549+0000" 2024-09-18T14:45:26.214 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:26.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:26.228 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:26.228 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.d") | .acting_primary' 2024-09-18T14:45:26.580 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:26.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=3 2024-09-18T14:45:26.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="3.d"' 2024-09-18T14:45:26.594 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=3.d 2024-09-18T14:45:26.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 3.d scrub 2024-09-18T14:45:26.732 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:26.732 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:26.733 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:46.729037+0000" 2024-09-18T14:45:26.733 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:26.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:26.745 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:26.745 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.f") | .acting_primary' 2024-09-18T14:45:27.059 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:27.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=3 2024-09-18T14:45:27.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="1.f"' 2024-09-18T14:45:27.071 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=1.f 2024-09-18T14:45:27.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 1.f scrub 2024-09-18T14:45:27.211 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:27.211 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:27.211 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:47.207691+0000" 2024-09-18T14:45:27.211 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:27.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2024-09-18T14:45:27.223 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph pg dump pgs --format=json 2024-09-18T14:45:27.223 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.c") | .acting_primary' 2024-09-18T14:45:27.590 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:27.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: primary=2 2024-09-18T14:45:27.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: eval 'pg="2.c"' 2024-09-18T14:45:27.602 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:89: TEST_recover_unexpected: pg=2.c 2024-09-18T14:45:27.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:90: TEST_recover_unexpected: ceph tell 2.c scrub 2024-09-18T14:45:27.742 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:27.743 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:45:27.743 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:43:47.739402+0000" 2024-09-18T14:45:27.743 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:27.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:93: TEST_recover_unexpected: ceph pg dump pgs 2024-09-18T14:45:28.074 INFO:tasks.workunit.client.0.smithi046.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:45:28.075 INFO:tasks.workunit.client.0.smithi046.stdout:3.9 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2024-09-18T14:36:52.956816+0000 44'61 46:74 [4,2,5] 4 [4,2,5] 4 0'0 2024-09-11T14:43:24.774712+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:54:50.198653+0000 0 0 2024-09-18T14:45:28.075 INFO:tasks.workunit.client.0.smithi046.stdout:1.b 65 0 0 0 0 532480 0 0 65 0 65 active+clean 2024-09-18T14:36:49.007446+0000 44'65 46:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-11T14:43:25.259507+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:46:34.190554+0000 0 0 2024-09-18T14:45:28.075 INFO:tasks.workunit.client.0.smithi046.stdout:2.8 66 0 0 0 0 540672 0 0 66 0 66 active+clean 2024-09-18T14:36:50.939099+0000 44'66 46:85 [0,2,1] 0 [0,2,1] 0 0'0 2024-09-11T14:43:25.731609+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:05:41.940007+0000 0 0 2024-09-18T14:45:28.075 INFO:tasks.workunit.client.0.smithi046.stdout:3.8 66 0 0 0 0 540672 0 0 66 0 66 active+clean 2024-09-18T14:36:52.955028+0000 44'66 46:79 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-11T14:43:26.201260+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:34:06.207298+0000 0 0 2024-09-18T14:45:28.075 INFO:tasks.workunit.client.0.smithi046.stdout:1.a 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2024-09-18T14:36:49.008053+0000 44'61 46:82 [1,2,4] 1 [1,2,4] 1 0'0 2024-09-11T14:43:26.694407+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:17:35.387520+0000 0 0 2024-09-18T14:45:28.075 INFO:tasks.workunit.client.0.smithi046.stdout:2.9 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2024-09-18T14:36:50.619265+0000 44'61 46:77 [4,1,0] 4 [4,1,0] 4 0'0 2024-09-11T14:43:27.166635+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:15:48.327074+0000 0 0 2024-09-18T14:45:28.075 INFO:tasks.workunit.client.0.smithi046.stdout:3.b 65 0 0 0 0 532480 0 0 65 0 65 active+clean 2024-09-18T14:36:53.000633+0000 44'65 46:78 [3,0,4] 3 [3,0,4] 3 0'0 2024-09-11T14:43:27.676050+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:01:51.561657+0000 0 0 2024-09-18T14:45:28.075 INFO:tasks.workunit.client.0.smithi046.stdout:1.9 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2024-09-18T14:36:48.700675+0000 44'61 46:82 [1,4,5] 1 [1,4,5] 1 0'0 2024-09-11T14:43:28.151548+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-20T02:01:25.774857+0000 0 0 2024-09-18T14:45:28.076 INFO:tasks.workunit.client.0.smithi046.stdout:2.a 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2024-09-18T14:36:50.479196+0000 44'61 46:79 [1,5,0] 1 [1,5,0] 1 0'0 2024-09-11T14:43:28.665056+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:41:50.153043+0000 0 0 2024-09-18T14:45:28.076 INFO:tasks.workunit.client.0.smithi046.stdout:3.a 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2024-09-18T14:36:52.955446+0000 44'61 46:74 [3,1,5] 3 [3,1,5] 3 0'0 2024-09-11T14:43:29.144417+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:06:01.652456+0000 0 0 2024-09-18T14:45:28.076 INFO:tasks.workunit.client.0.smithi046.stdout:1.8 66 0 0 0 0 540672 0 0 66 0 66 active+clean 2024-09-18T14:36:48.698901+0000 44'66 46:86 [5,4,3] 5 [5,4,3] 5 0'0 2024-09-11T14:43:29.615673+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:25:38.543673+0000 0 0 2024-09-18T14:45:28.076 INFO:tasks.workunit.client.0.smithi046.stdout:2.b 65 0 0 0 0 532480 0 0 65 0 65 active+clean 2024-09-18T14:36:50.937469+0000 44'65 46:81 [3,2,5] 3 [3,2,5] 3 0'0 2024-09-11T14:43:30.097882+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:31:57.118081+0000 0 0 2024-09-18T14:45:28.076 INFO:tasks.workunit.client.0.smithi046.stdout:3.5 73 0 0 0 0 598016 0 0 73 0 73 active+clean 2024-09-18T14:36:52.953974+0000 44'73 46:86 [5,3,4] 5 [5,3,4] 5 0'0 2024-09-11T14:43:30.580906+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:46:48.338536+0000 0 0 2024-09-18T14:45:28.076 INFO:tasks.workunit.client.0.smithi046.stdout:1.7 53 0 0 0 0 434176 0 0 53 0 53 active+clean 2024-09-18T14:36:48.661525+0000 44'53 46:75 [1,3,0] 1 [1,3,0] 1 0'0 2024-09-11T14:43:31.061428+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:49:40.680653+0000 0 0 2024-09-18T14:45:28.076 INFO:tasks.workunit.client.0.smithi046.stdout:2.4 51 0 0 0 0 417792 0 0 51 0 51 active+clean 2024-09-18T14:36:50.937825+0000 44'51 46:69 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-11T14:43:31.535083+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:59:06.262828+0000 0 0 2024-09-18T14:45:28.076 INFO:tasks.workunit.client.0.smithi046.stdout:3.4 51 0 0 0 0 417792 0 0 51 0 51 active+clean 2024-09-18T14:36:52.954243+0000 44'51 46:65 [1,2,4] 1 [1,2,4] 1 0'0 2024-09-11T14:43:32.019342+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:28:24.899070+0000 0 0 2024-09-18T14:45:28.076 INFO:tasks.workunit.client.0.smithi046.stdout:1.6 64 0 0 0 0 524288 0 0 64 0 64 active+clean 2024-09-18T14:36:48.871192+0000 44'64 46:84 [3,4,2] 3 [3,4,2] 3 0'0 2024-09-11T14:43:32.498437+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T22:48:20.771546+0000 0 0 2024-09-18T14:45:28.076 INFO:tasks.workunit.client.0.smithi046.stdout:2.5 73 0 0 0 0 598016 0 0 73 0 73 active+clean 2024-09-18T14:36:50.618142+0000 44'73 46:89 [3,4,0] 3 [3,4,0] 3 0'0 2024-09-11T14:43:32.990299+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:04:30.830160+0000 0 0 2024-09-18T14:45:28.076 INFO:tasks.workunit.client.0.smithi046.stdout:3.7 53 0 0 0 0 434176 0 0 53 0 53 active+clean 2024-09-18T14:36:52.998666+0000 44'53 46:66 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-11T14:43:33.461488+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:03:15.231745+0000 0 0 2024-09-18T14:45:28.076 INFO:tasks.workunit.client.0.smithi046.stdout:1.5 73 0 0 0 0 598016 0 0 73 0 73 active+clean 2024-09-18T14:36:48.872088+0000 44'73 46:93 [4,2,5] 4 [4,2,5] 4 0'0 2024-09-11T14:43:33.930364+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:48:03.126183+0000 0 0 2024-09-18T14:45:28.077 INFO:tasks.workunit.client.0.smithi046.stdout:2.6 64 0 0 0 0 524288 0 0 64 0 64 active+clean 2024-09-18T14:36:50.618738+0000 44'64 46:82 [1,0,4] 1 [1,0,4] 1 0'0 2024-09-11T14:43:34.430687+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:09:48.734576+0000 0 0 2024-09-18T14:45:28.077 INFO:tasks.workunit.client.0.smithi046.stdout:3.6 64 0 0 0 0 524288 0 0 64 0 64 active+clean 2024-09-18T14:36:52.998021+0000 44'64 46:78 [0,1,4] 0 [0,1,4] 0 0'0 2024-09-11T14:43:34.909707+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:57:19.264216+0000 0 0 2024-09-18T14:45:28.077 INFO:tasks.workunit.client.0.smithi046.stdout:1.4 51 0 0 0 0 417792 0 0 51 0 51 active+clean 2024-09-18T14:36:48.662483+0000 44'51 46:71 [3,0,1] 3 [3,0,1] 3 0'0 2024-09-11T14:43:35.407291+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:13:16.097178+0000 0 0 2024-09-18T14:45:28.077 INFO:tasks.workunit.client.0.smithi046.stdout:2.7 53 0 0 0 0 434176 0 0 53 0 53 active+clean 2024-09-18T14:36:50.937073+0000 44'53 46:69 [3,0,2] 3 [3,0,2] 3 0'0 2024-09-11T14:43:35.879954+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:38:47.644254+0000 0 0 2024-09-18T14:45:28.077 INFO:tasks.workunit.client.0.smithi046.stdout:3.1 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2024-09-18T14:36:52.954138+0000 44'67 46:80 [4,3,5] 4 [4,3,5] 4 0'0 2024-09-11T14:43:36.441821+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:18:23.308670+0000 0 0 2024-09-18T14:45:28.077 INFO:tasks.workunit.client.0.smithi046.stdout:1.3 54 0 0 0 0 442368 0 0 54 0 54 active+clean 2024-09-18T14:36:48.871361+0000 44'54 46:74 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-11T14:43:36.928924+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:57:43.630442+0000 0 0 2024-09-18T14:45:28.077 INFO:tasks.workunit.client.0.smithi046.stdout:2.0 72 0 0 0 0 589824 0 0 72 0 72 active+clean 2024-09-18T14:36:50.621219+0000 44'72 46:88 [3,5,1] 3 [3,5,1] 3 0'0 2024-09-11T14:43:37.417204+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-20T02:06:16.586258+0000 0 0 2024-09-18T14:45:28.077 INFO:tasks.workunit.client.0.smithi046.stdout:3.0 72 0 0 0 0 589824 0 0 72 0 72 active+clean 2024-09-18T14:36:52.956275+0000 44'72 46:86 [1,2,4] 1 [1,2,4] 1 0'0 2024-09-11T14:43:37.894982+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:31:16.804864+0000 0 0 2024-09-18T14:45:28.077 INFO:tasks.workunit.client.0.smithi046.stdout:1.2 55 0 0 0 0 450560 0 0 55 0 55 active+clean 2024-09-18T14:36:48.699754+0000 44'55 46:78 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-11T14:43:38.436840+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:01:05.968369+0000 0 0 2024-09-18T14:45:28.077 INFO:tasks.workunit.client.0.smithi046.stdout:2.1 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2024-09-18T14:36:51.681450+0000 44'67 46:86 [2,5,0] 2 [2,5,0] 2 0'0 2024-09-11T14:43:38.913415+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:23:12.513074+0000 0 0 2024-09-18T14:45:28.077 INFO:tasks.workunit.client.0.smithi046.stdout:3.e 69 0 0 0 0 565248 0 0 69 0 69 active+clean 2024-09-18T14:36:52.918261+0000 44'69 46:83 [2,4,1] 2 [2,4,1] 2 0'0 2024-09-11T14:43:39.401311+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:20:54.223154+0000 0 0 2024-09-18T14:45:28.077 INFO:tasks.workunit.client.0.smithi046.stdout:2.f 62 0 0 0 0 507904 0 0 62 0 62 active+clean 2024-09-18T14:36:50.617118+0000 44'62 46:81 [0,5,4] 0 [0,5,4] 0 0'0 2024-09-11T14:43:39.878926+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:48:35.621383+0000 0 0 2024-09-18T14:45:28.078 INFO:tasks.workunit.client.0.smithi046.stdout:1.c 60 0 0 0 0 491520 0 0 60 0 60 active+clean 2024-09-18T14:36:48.593385+0000 44'60 46:80 [3,0,4] 3 [3,0,4] 3 0'0 2024-09-11T14:43:40.347430+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T23:04:03.428915+0000 0 0 2024-09-18T14:45:28.078 INFO:tasks.workunit.client.0.smithi046.stdout:3.3 54 0 0 0 0 442368 0 0 54 0 54 active+clean 2024-09-18T14:36:52.998753+0000 44'54 46:67 [4,0,5] 4 [4,0,5] 4 0'0 2024-09-11T14:43:40.831017+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:06:31.626467+0000 0 0 2024-09-18T14:45:28.078 INFO:tasks.workunit.client.0.smithi046.stdout:1.1 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2024-09-18T14:36:48.697833+0000 44'67 46:87 [5,0,4] 5 [5,0,4] 5 0'0 2024-09-11T14:43:41.345867+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:42:04.100999+0000 0 0 2024-09-18T14:45:28.078 INFO:tasks.workunit.client.0.smithi046.stdout:2.2 55 0 0 0 0 450560 0 0 55 0 55 active+clean 2024-09-18T14:36:50.622984+0000 44'55 44:70 [5,1,0] 5 [5,1,0] 5 0'0 2024-09-18T14:36:48.785595+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:28:55.204743+0000 0 0 2024-09-18T14:45:28.078 INFO:tasks.workunit.client.0.smithi046.stdout:3.2 55 0 0 0 0 450560 0 0 55 0 55 active+clean 2024-09-18T14:36:52.997928+0000 44'55 45:67 [3,5,0] 3 [3,5,0] 3 0'0 2024-09-18T14:36:51.279535+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:52:49.736896+0000 0 0 2024-09-18T14:45:28.078 INFO:tasks.workunit.client.0.smithi046.stdout:1.0 72 0 0 0 0 589824 0 0 72 0 72 active+clean 2024-09-18T14:36:49.007560+0000 44'72 44:92 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:36:46.926781+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T22:36:11.932207+0000 0 0 2024-09-18T14:45:28.078 INFO:tasks.workunit.client.0.smithi046.stdout:2.3 54 0 0 0 0 442368 0 0 54 0 54 active+clean 2024-09-18T14:36:50.624348+0000 44'54 44:69 [5,4,1] 5 [5,4,1] 5 0'0 2024-09-18T14:36:48.785595+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-20T02:23:11.701332+0000 0 0 2024-09-18T14:45:28.078 INFO:tasks.workunit.client.0.smithi046.stdout:3.f 62 0 0 0 0 507904 0 0 62 0 62 active+clean 2024-09-18T14:36:52.917739+0000 44'62 46:76 [0,1,4] 0 [0,1,4] 0 0'0 2024-09-11T14:43:43.739831+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:46:00.909866+0000 0 0 2024-09-18T14:45:28.078 INFO:tasks.workunit.client.0.smithi046.stdout:2.e 69 0 0 0 0 565248 0 0 69 0 69 active+clean 2024-09-18T14:36:51.681573+0000 44'69 46:88 [2,3,5] 2 [2,3,5] 2 0'0 2024-09-11T14:43:44.209216+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:46:51.645906+0000 0 0 2024-09-18T14:45:28.078 INFO:tasks.workunit.client.0.smithi046.stdout:1.d 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2024-09-18T14:36:48.081685+0000 44'67 45:89 [2,4,1] 2 [2,4,1] 2 0'0 2024-09-18T14:36:46.926781+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:20:04.204830+0000 0 0 2024-09-18T14:45:28.078 INFO:tasks.workunit.client.0.smithi046.stdout:3.c 60 0 0 0 0 491520 0 0 60 0 60 active+clean 2024-09-18T14:36:52.954220+0000 44'60 44:72 [5,3,1] 5 [5,3,1] 5 0'0 2024-09-18T14:36:51.279535+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T23:05:42.924123+0000 0 0 2024-09-18T14:45:28.078 INFO:tasks.workunit.client.0.smithi046.stdout:2.d 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2024-09-18T14:36:50.623874+0000 44'67 44:82 [5,1,3] 5 [5,1,3] 5 0'0 2024-09-18T14:36:48.785595+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:03:59.303765+0000 0 0 2024-09-18T14:45:28.078 INFO:tasks.workunit.client.0.smithi046.stdout:1.e 69 0 0 0 0 565248 0 0 69 0 69 active+clean 2024-09-18T14:36:49.007425+0000 44'69 44:89 [1,4,2] 1 [1,4,2] 1 0'0 2024-09-18T14:36:46.926781+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:54:04.420111+0000 0 0 2024-09-18T14:45:28.079 INFO:tasks.workunit.client.0.smithi046.stdout:3.d 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2024-09-18T14:36:52.998476+0000 44'67 45:79 [3,1,0] 3 [3,1,0] 3 0'0 2024-09-18T14:36:51.279535+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:16:20.566519+0000 0 0 2024-09-18T14:45:28.079 INFO:tasks.workunit.client.0.smithi046.stdout:1.f 62 0 0 0 0 507904 0 0 62 0 62 active+clean 2024-09-18T14:36:48.593494+0000 44'62 45:81 [3,0,4] 3 [3,0,4] 3 0'0 2024-09-18T14:36:46.926781+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:25:09.813452+0000 0 0 2024-09-18T14:45:28.079 INFO:tasks.workunit.client.0.smithi046.stdout:2.c 60 0 0 0 0 491520 0 0 60 0 60 active+clean 2024-09-18T14:36:51.681038+0000 44'60 45:78 [2,0,5] 2 [2,0,5] 2 0'0 2024-09-18T14:36:48.785595+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:06:14.605209+0000 0 0 2024-09-18T14:45:28.079 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-09-18T14:45:28.079 INFO:tasks.workunit.client.0.smithi046.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:45:28.079 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:28.088 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:95: TEST_recover_unexpected: jq .osd_max_scrubs 2024-09-18T14:45:28.089 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:95: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:45:28.089 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:45:28.089 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:45:28.090 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:28.090 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:28.090 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:28.090 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:45:28.091 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:95: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:28.091 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:95: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:45:28.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:95: TEST_recover_unexpected: max=4 2024-09-18T14:45:28.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:96: TEST_recover_unexpected: '[' 4 '!=' 4 ']' 2024-09-18T14:45:28.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:101: TEST_recover_unexpected: ceph osd unset noscrub 2024-09-18T14:45:28.665 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is unset 2024-09-18T14:45:28.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:103: TEST_recover_unexpected: ok=false 2024-09-18T14:45:28.677 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:104: TEST_recover_unexpected: seq 0 300 2024-09-18T14:45:28.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:104: TEST_recover_unexpected: for i in $(seq 0 300) 2024-09-18T14:45:28.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:106: TEST_recover_unexpected: ceph pg dump pgs 2024-09-18T14:45:29.025 INFO:tasks.workunit.client.0.smithi046.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:45:29.025 INFO:tasks.workunit.client.0.smithi046.stdout:3.9 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2024-09-18T14:36:52.956816+0000 44'61 46:74 [4,2,5] 4 [4,2,5] 4 0'0 2024-09-11T14:43:24.774712+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:54:50.198653+0000 0 0 2024-09-18T14:45:29.026 INFO:tasks.workunit.client.0.smithi046.stdout:1.b 65 0 0 0 0 532480 0 0 65 0 65 active+clean 2024-09-18T14:36:49.007446+0000 44'65 46:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-11T14:43:25.259507+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:46:34.190554+0000 0 0 2024-09-18T14:45:29.026 INFO:tasks.workunit.client.0.smithi046.stdout:2.8 66 0 0 0 0 540672 0 0 66 0 66 active+clean 2024-09-18T14:36:50.939099+0000 44'66 46:85 [0,2,1] 0 [0,2,1] 0 0'0 2024-09-11T14:43:25.731609+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:05:41.940007+0000 0 0 2024-09-18T14:45:29.026 INFO:tasks.workunit.client.0.smithi046.stdout:3.8 66 0 0 0 0 540672 0 0 66 0 66 active+clean 2024-09-18T14:36:52.955028+0000 44'66 46:79 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-11T14:43:26.201260+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:34:06.207298+0000 0 0 2024-09-18T14:45:29.026 INFO:tasks.workunit.client.0.smithi046.stdout:1.a 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2024-09-18T14:36:49.008053+0000 44'61 46:82 [1,2,4] 1 [1,2,4] 1 0'0 2024-09-11T14:43:26.694407+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:17:35.387520+0000 0 0 2024-09-18T14:45:29.026 INFO:tasks.workunit.client.0.smithi046.stdout:2.9 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2024-09-18T14:36:50.619265+0000 44'61 46:77 [4,1,0] 4 [4,1,0] 4 0'0 2024-09-11T14:43:27.166635+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:15:48.327074+0000 0 0 2024-09-18T14:45:29.026 INFO:tasks.workunit.client.0.smithi046.stdout:3.b 65 0 0 0 0 532480 0 0 65 0 65 active+clean 2024-09-18T14:36:53.000633+0000 44'65 46:78 [3,0,4] 3 [3,0,4] 3 0'0 2024-09-11T14:43:27.676050+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:01:51.561657+0000 0 0 2024-09-18T14:45:29.026 INFO:tasks.workunit.client.0.smithi046.stdout:1.9 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2024-09-18T14:36:48.700675+0000 44'61 46:82 [1,4,5] 1 [1,4,5] 1 0'0 2024-09-11T14:43:28.151548+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-20T02:01:25.774857+0000 0 0 2024-09-18T14:45:29.026 INFO:tasks.workunit.client.0.smithi046.stdout:2.a 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2024-09-18T14:36:50.479196+0000 44'61 46:79 [1,5,0] 1 [1,5,0] 1 0'0 2024-09-11T14:43:28.665056+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:41:50.153043+0000 0 0 2024-09-18T14:45:29.026 INFO:tasks.workunit.client.0.smithi046.stdout:3.a 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2024-09-18T14:36:52.955446+0000 44'61 46:74 [3,1,5] 3 [3,1,5] 3 0'0 2024-09-11T14:43:29.144417+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:06:01.652456+0000 0 0 2024-09-18T14:45:29.026 INFO:tasks.workunit.client.0.smithi046.stdout:1.8 66 0 0 0 0 540672 0 0 66 0 66 active+clean 2024-09-18T14:36:48.698901+0000 44'66 46:86 [5,4,3] 5 [5,4,3] 5 0'0 2024-09-11T14:43:29.615673+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:25:38.543673+0000 0 0 2024-09-18T14:45:29.026 INFO:tasks.workunit.client.0.smithi046.stdout:2.b 65 0 0 0 0 532480 0 0 65 0 65 active+clean 2024-09-18T14:36:50.937469+0000 44'65 46:81 [3,2,5] 3 [3,2,5] 3 0'0 2024-09-11T14:43:30.097882+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:31:57.118081+0000 0 0 2024-09-18T14:45:29.026 INFO:tasks.workunit.client.0.smithi046.stdout:3.5 73 0 0 0 0 598016 0 0 73 0 73 active+clean 2024-09-18T14:36:52.953974+0000 44'73 46:86 [5,3,4] 5 [5,3,4] 5 0'0 2024-09-11T14:43:30.580906+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:46:48.338536+0000 0 0 2024-09-18T14:45:29.027 INFO:tasks.workunit.client.0.smithi046.stdout:1.7 53 0 0 0 0 434176 0 0 53 0 53 active+clean 2024-09-18T14:36:48.661525+0000 44'53 46:75 [1,3,0] 1 [1,3,0] 1 0'0 2024-09-11T14:43:31.061428+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:49:40.680653+0000 0 0 2024-09-18T14:45:29.027 INFO:tasks.workunit.client.0.smithi046.stdout:2.4 51 0 0 0 0 417792 0 0 51 0 51 active+clean 2024-09-18T14:36:50.937825+0000 44'51 46:69 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-11T14:43:31.535083+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:59:06.262828+0000 0 0 2024-09-18T14:45:29.027 INFO:tasks.workunit.client.0.smithi046.stdout:3.4 51 0 0 0 0 417792 0 0 51 0 51 active+clean 2024-09-18T14:36:52.954243+0000 44'51 46:65 [1,2,4] 1 [1,2,4] 1 0'0 2024-09-11T14:43:32.019342+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:28:24.899070+0000 0 0 2024-09-18T14:45:29.027 INFO:tasks.workunit.client.0.smithi046.stdout:1.6 64 0 0 0 0 524288 0 0 64 0 64 active+clean 2024-09-18T14:36:48.871192+0000 44'64 46:84 [3,4,2] 3 [3,4,2] 3 0'0 2024-09-11T14:43:32.498437+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T22:48:20.771546+0000 0 0 2024-09-18T14:45:29.027 INFO:tasks.workunit.client.0.smithi046.stdout:2.5 73 0 0 0 0 598016 0 0 73 0 73 active+clean 2024-09-18T14:36:50.618142+0000 44'73 46:89 [3,4,0] 3 [3,4,0] 3 0'0 2024-09-11T14:43:32.990299+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:04:30.830160+0000 0 0 2024-09-18T14:45:29.027 INFO:tasks.workunit.client.0.smithi046.stdout:3.7 53 0 0 0 0 434176 0 0 53 0 53 active+clean 2024-09-18T14:36:52.998666+0000 44'53 46:66 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-11T14:43:33.461488+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:03:15.231745+0000 0 0 2024-09-18T14:45:29.027 INFO:tasks.workunit.client.0.smithi046.stdout:1.5 73 0 0 0 0 598016 0 0 73 0 73 active+clean 2024-09-18T14:36:48.872088+0000 44'73 46:93 [4,2,5] 4 [4,2,5] 4 0'0 2024-09-11T14:43:33.930364+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:48:03.126183+0000 0 0 2024-09-18T14:45:29.027 INFO:tasks.workunit.client.0.smithi046.stdout:2.6 64 0 0 0 0 524288 0 0 64 0 64 active+clean 2024-09-18T14:36:50.618738+0000 44'64 46:82 [1,0,4] 1 [1,0,4] 1 0'0 2024-09-11T14:43:34.430687+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:09:48.734576+0000 0 0 2024-09-18T14:45:29.027 INFO:tasks.workunit.client.0.smithi046.stdout:3.6 64 0 0 0 0 524288 0 0 64 0 64 active+clean 2024-09-18T14:36:52.998021+0000 44'64 46:78 [0,1,4] 0 [0,1,4] 0 0'0 2024-09-11T14:43:34.909707+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:57:19.264216+0000 0 0 2024-09-18T14:45:29.027 INFO:tasks.workunit.client.0.smithi046.stdout:1.4 51 0 0 0 0 417792 0 0 51 0 51 active+clean 2024-09-18T14:36:48.662483+0000 44'51 46:71 [3,0,1] 3 [3,0,1] 3 0'0 2024-09-11T14:43:35.407291+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:13:16.097178+0000 0 0 2024-09-18T14:45:29.027 INFO:tasks.workunit.client.0.smithi046.stdout:2.7 53 0 0 0 0 434176 0 0 53 0 53 active+clean 2024-09-18T14:36:50.937073+0000 44'53 46:69 [3,0,2] 3 [3,0,2] 3 0'0 2024-09-11T14:43:35.879954+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:38:47.644254+0000 0 0 2024-09-18T14:45:29.027 INFO:tasks.workunit.client.0.smithi046.stdout:3.1 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2024-09-18T14:36:52.954138+0000 44'67 46:80 [4,3,5] 4 [4,3,5] 4 0'0 2024-09-11T14:43:36.441821+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:18:23.308670+0000 0 0 2024-09-18T14:45:29.027 INFO:tasks.workunit.client.0.smithi046.stdout:1.3 54 0 0 0 0 442368 0 0 54 0 54 active+clean 2024-09-18T14:36:48.871361+0000 44'54 46:74 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-11T14:43:36.928924+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:57:43.630442+0000 0 0 2024-09-18T14:45:29.028 INFO:tasks.workunit.client.0.smithi046.stdout:2.0 72 0 0 0 0 589824 0 0 72 0 72 active+clean 2024-09-18T14:36:50.621219+0000 44'72 46:88 [3,5,1] 3 [3,5,1] 3 0'0 2024-09-11T14:43:37.417204+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-20T02:06:16.586258+0000 0 0 2024-09-18T14:45:29.028 INFO:tasks.workunit.client.0.smithi046.stdout:3.0 72 0 0 0 0 589824 0 0 72 0 72 active+clean 2024-09-18T14:36:52.956275+0000 44'72 46:86 [1,2,4] 1 [1,2,4] 1 0'0 2024-09-11T14:43:37.894982+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:31:16.804864+0000 0 0 2024-09-18T14:45:29.028 INFO:tasks.workunit.client.0.smithi046.stdout:1.2 55 0 0 0 0 450560 0 0 55 0 55 active+clean 2024-09-18T14:36:48.699754+0000 44'55 46:78 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-11T14:43:38.436840+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:01:05.968369+0000 0 0 2024-09-18T14:45:29.028 INFO:tasks.workunit.client.0.smithi046.stdout:2.1 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2024-09-18T14:36:51.681450+0000 44'67 46:86 [2,5,0] 2 [2,5,0] 2 0'0 2024-09-11T14:43:38.913415+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:23:12.513074+0000 0 0 2024-09-18T14:45:29.028 INFO:tasks.workunit.client.0.smithi046.stdout:3.e 69 0 0 0 0 565248 0 0 69 0 69 active+clean 2024-09-18T14:36:52.918261+0000 44'69 46:83 [2,4,1] 2 [2,4,1] 2 0'0 2024-09-11T14:43:39.401311+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:20:54.223154+0000 0 0 2024-09-18T14:45:29.028 INFO:tasks.workunit.client.0.smithi046.stdout:2.f 62 0 0 0 0 507904 0 0 62 0 62 active+clean 2024-09-18T14:36:50.617118+0000 44'62 46:81 [0,5,4] 0 [0,5,4] 0 0'0 2024-09-11T14:43:39.878926+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:48:35.621383+0000 0 0 2024-09-18T14:45:29.028 INFO:tasks.workunit.client.0.smithi046.stdout:1.c 60 0 0 0 0 491520 0 0 60 0 60 active+clean 2024-09-18T14:36:48.593385+0000 44'60 46:80 [3,0,4] 3 [3,0,4] 3 0'0 2024-09-11T14:43:40.347430+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T23:04:03.428915+0000 0 0 2024-09-18T14:45:29.028 INFO:tasks.workunit.client.0.smithi046.stdout:3.3 54 0 0 0 0 442368 0 0 54 0 54 active+clean 2024-09-18T14:36:52.998753+0000 44'54 46:67 [4,0,5] 4 [4,0,5] 4 0'0 2024-09-11T14:43:40.831017+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:06:31.626467+0000 0 0 2024-09-18T14:45:29.028 INFO:tasks.workunit.client.0.smithi046.stdout:1.1 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2024-09-18T14:36:48.697833+0000 44'67 46:87 [5,0,4] 5 [5,0,4] 5 0'0 2024-09-11T14:43:41.345867+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:42:04.100999+0000 0 0 2024-09-18T14:45:29.028 INFO:tasks.workunit.client.0.smithi046.stdout:2.2 55 0 0 0 0 450560 0 0 55 0 55 active+clean 2024-09-18T14:36:50.622984+0000 44'55 46:71 [5,1,0] 5 [5,1,0] 5 0'0 2024-09-11T14:43:41.813881+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:28:55.204743+0000 0 0 2024-09-18T14:45:29.028 INFO:tasks.workunit.client.0.smithi046.stdout:3.2 55 0 0 0 0 450560 0 0 55 0 55 active+clean 2024-09-18T14:36:52.997928+0000 44'55 46:68 [3,5,0] 3 [3,5,0] 3 0'0 2024-09-11T14:43:42.300039+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:52:49.736896+0000 0 0 2024-09-18T14:45:29.028 INFO:tasks.workunit.client.0.smithi046.stdout:1.0 72 0 0 0 0 589824 0 0 72 0 72 active+clean 2024-09-18T14:36:49.007560+0000 44'72 46:93 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-11T14:43:42.785839+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T22:36:11.932207+0000 0 0 2024-09-18T14:45:29.029 INFO:tasks.workunit.client.0.smithi046.stdout:2.3 54 0 0 0 0 442368 0 0 54 0 54 active+clean 2024-09-18T14:36:50.624348+0000 44'54 46:70 [5,4,1] 5 [5,4,1] 5 0'0 2024-09-11T14:43:43.262049+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-20T02:23:11.701332+0000 0 0 2024-09-18T14:45:29.029 INFO:tasks.workunit.client.0.smithi046.stdout:3.f 62 0 0 0 0 507904 0 0 62 0 62 active+clean 2024-09-18T14:36:52.917739+0000 44'62 46:76 [0,1,4] 0 [0,1,4] 0 0'0 2024-09-11T14:43:43.739831+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:46:00.909866+0000 0 0 2024-09-18T14:45:29.029 INFO:tasks.workunit.client.0.smithi046.stdout:2.e 69 0 0 0 0 565248 0 0 69 0 69 active+clean 2024-09-18T14:36:51.681573+0000 44'69 46:88 [2,3,5] 2 [2,3,5] 2 0'0 2024-09-11T14:43:44.209216+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:46:51.645906+0000 0 0 2024-09-18T14:45:29.029 INFO:tasks.workunit.client.0.smithi046.stdout:1.d 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2024-09-18T14:36:48.081685+0000 44'67 45:89 [2,4,1] 2 [2,4,1] 2 0'0 2024-09-18T14:36:46.926781+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:20:04.204830+0000 0 0 2024-09-18T14:45:29.029 INFO:tasks.workunit.client.0.smithi046.stdout:3.c 60 0 0 0 0 491520 0 0 60 0 60 active+clean 2024-09-18T14:36:52.954220+0000 44'60 46:73 [5,3,1] 5 [5,3,1] 5 0'0 2024-09-11T14:43:45.208366+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T23:05:42.924123+0000 0 0 2024-09-18T14:45:29.029 INFO:tasks.workunit.client.0.smithi046.stdout:2.d 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2024-09-18T14:36:50.623874+0000 44'67 46:83 [5,1,3] 5 [5,1,3] 5 0'0 2024-09-11T14:43:45.730754+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:03:59.303765+0000 0 0 2024-09-18T14:45:29.029 INFO:tasks.workunit.client.0.smithi046.stdout:1.e 69 0 0 0 0 565248 0 0 69 0 69 active+clean 2024-09-18T14:36:49.007425+0000 44'69 46:90 [1,4,2] 1 [1,4,2] 1 0'0 2024-09-11T14:43:46.210549+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:54:04.420111+0000 0 0 2024-09-18T14:45:29.029 INFO:tasks.workunit.client.0.smithi046.stdout:3.d 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2024-09-18T14:36:52.998476+0000 44'67 46:80 [3,1,0] 3 [3,1,0] 3 0'0 2024-09-11T14:43:46.729037+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:16:20.566519+0000 0 0 2024-09-18T14:45:29.029 INFO:tasks.workunit.client.0.smithi046.stdout:1.f 62 0 0 0 0 507904 0 0 62 0 62 active+clean 2024-09-18T14:36:48.593494+0000 44'62 45:81 [3,0,4] 3 [3,0,4] 3 0'0 2024-09-18T14:36:46.926781+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:25:09.813452+0000 0 0 2024-09-18T14:45:29.029 INFO:tasks.workunit.client.0.smithi046.stdout:2.c 60 0 0 0 0 491520 0 0 60 0 60 active+clean 2024-09-18T14:36:51.681038+0000 44'60 45:78 [2,0,5] 2 [2,0,5] 2 0'0 2024-09-18T14:36:48.785595+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:06:14.605209+0000 0 0 2024-09-18T14:45:29.029 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-09-18T14:45:29.029 INFO:tasks.workunit.client.0.smithi046.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:45:29.030 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:29.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:107: TEST_recover_unexpected: ceph pg dump pgs 2024-09-18T14:45:29.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:107: TEST_recover_unexpected: grep +scrubbing 2024-09-18T14:45:29.348 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:29.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:111: TEST_recover_unexpected: sleep 1 2024-09-18T14:45:30.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:104: TEST_recover_unexpected: for i in $(seq 0 300) 2024-09-18T14:45:30.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:106: TEST_recover_unexpected: ceph pg dump pgs 2024-09-18T14:45:30.689 INFO:tasks.workunit.client.0.smithi046.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:45:30.689 INFO:tasks.workunit.client.0.smithi046.stdout:3.9 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2024-09-18T14:36:52.956816+0000 44'61 46:75 [4,2,5] 4 [4,2,5] 4 0'0 2024-09-11T14:43:24.774712+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 queued for scrub 0 0 2024-09-18T14:45:30.689 INFO:tasks.workunit.client.0.smithi046.stdout:1.b 65 0 0 0 0 532480 0 0 65 0 65 active+clean 2024-09-18T14:36:49.007446+0000 44'65 46:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-11T14:43:25.259507+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:46:34.190554+0000 0 0 2024-09-18T14:45:30.689 INFO:tasks.workunit.client.0.smithi046.stdout:2.8 66 0 0 0 0 540672 0 0 66 0 66 active+clean 2024-09-18T14:36:50.939099+0000 44'66 46:86 [0,2,1] 0 [0,2,1] 0 0'0 2024-09-11T14:43:25.731609+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 queued for scrub 0 0 2024-09-18T14:45:30.689 INFO:tasks.workunit.client.0.smithi046.stdout:3.8 66 0 0 0 0 540672 0 0 66 0 66 active+clean 2024-09-18T14:36:52.955028+0000 44'66 46:80 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-11T14:43:26.201260+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 queued for scrub 0 0 2024-09-18T14:45:30.690 INFO:tasks.workunit.client.0.smithi046.stdout:1.a 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2024-09-18T14:36:49.008053+0000 44'61 46:82 [1,2,4] 1 [1,2,4] 1 0'0 2024-09-11T14:43:26.694407+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:17:35.387520+0000 0 0 2024-09-18T14:45:30.690 INFO:tasks.workunit.client.0.smithi046.stdout:2.9 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2024-09-18T14:36:50.619265+0000 44'61 46:78 [4,1,0] 4 [4,1,0] 4 0'0 2024-09-11T14:43:27.166635+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 queued for scrub 0 0 2024-09-18T14:45:30.690 INFO:tasks.workunit.client.0.smithi046.stdout:3.b 65 0 0 0 0 532480 0 0 65 0 65 active+clean 2024-09-18T14:36:53.000633+0000 44'65 46:78 [3,0,4] 3 [3,0,4] 3 0'0 2024-09-11T14:43:27.676050+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:01:51.561657+0000 0 0 2024-09-18T14:45:30.690 INFO:tasks.workunit.client.0.smithi046.stdout:1.9 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2024-09-18T14:36:48.700675+0000 44'61 46:82 [1,4,5] 1 [1,4,5] 1 0'0 2024-09-11T14:43:28.151548+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-20T02:01:25.774857+0000 0 0 2024-09-18T14:45:30.690 INFO:tasks.workunit.client.0.smithi046.stdout:2.a 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2024-09-18T14:36:50.479196+0000 44'61 46:79 [1,5,0] 1 [1,5,0] 1 0'0 2024-09-11T14:43:28.665056+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:41:50.153043+0000 0 0 2024-09-18T14:45:30.690 INFO:tasks.workunit.client.0.smithi046.stdout:3.a 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2024-09-18T14:36:52.955446+0000 44'61 46:74 [3,1,5] 3 [3,1,5] 3 0'0 2024-09-11T14:43:29.144417+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:06:01.652456+0000 0 0 2024-09-18T14:45:30.690 INFO:tasks.workunit.client.0.smithi046.stdout:1.8 66 0 0 0 0 540672 0 0 66 0 66 active+clean 2024-09-18T14:36:48.698901+0000 44'66 46:86 [5,4,3] 5 [5,4,3] 5 0'0 2024-09-11T14:43:29.615673+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:25:38.543673+0000 0 0 2024-09-18T14:45:30.690 INFO:tasks.workunit.client.0.smithi046.stdout:2.b 65 0 0 0 0 532480 0 0 65 0 65 active+clean 2024-09-18T14:36:50.937469+0000 44'65 46:81 [3,2,5] 3 [3,2,5] 3 0'0 2024-09-11T14:43:30.097882+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:31:57.118081+0000 0 0 2024-09-18T14:45:30.690 INFO:tasks.workunit.client.0.smithi046.stdout:3.5 73 0 0 0 0 598016 0 0 73 0 73 active+clean 2024-09-18T14:36:52.953974+0000 44'73 46:86 [5,3,4] 5 [5,3,4] 5 0'0 2024-09-11T14:43:30.580906+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:46:48.338536+0000 0 0 2024-09-18T14:45:30.690 INFO:tasks.workunit.client.0.smithi046.stdout:1.7 53 0 0 0 0 434176 0 0 53 0 53 active+clean 2024-09-18T14:36:48.661525+0000 44'53 46:75 [1,3,0] 1 [1,3,0] 1 0'0 2024-09-11T14:43:31.061428+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:49:40.680653+0000 0 0 2024-09-18T14:45:30.690 INFO:tasks.workunit.client.0.smithi046.stdout:2.4 51 0 0 0 0 417792 0 0 51 0 51 active+clean 2024-09-18T14:36:50.937825+0000 44'51 46:69 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-11T14:43:31.535083+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:59:06.262828+0000 0 0 2024-09-18T14:45:30.691 INFO:tasks.workunit.client.0.smithi046.stdout:3.4 51 0 0 0 0 417792 0 0 51 0 51 active+clean 2024-09-18T14:36:52.954243+0000 44'51 46:65 [1,2,4] 1 [1,2,4] 1 0'0 2024-09-11T14:43:32.019342+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:28:24.899070+0000 0 0 2024-09-18T14:45:30.691 INFO:tasks.workunit.client.0.smithi046.stdout:1.6 64 0 0 0 0 524288 0 0 64 0 64 active+clean 2024-09-18T14:36:48.871192+0000 44'64 46:84 [3,4,2] 3 [3,4,2] 3 0'0 2024-09-11T14:43:32.498437+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T22:48:20.771546+0000 0 0 2024-09-18T14:45:30.691 INFO:tasks.workunit.client.0.smithi046.stdout:2.5 73 0 0 0 0 598016 0 0 73 0 73 active+clean 2024-09-18T14:36:50.618142+0000 44'73 46:89 [3,4,0] 3 [3,4,0] 3 0'0 2024-09-11T14:43:32.990299+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:04:30.830160+0000 0 0 2024-09-18T14:45:30.691 INFO:tasks.workunit.client.0.smithi046.stdout:3.7 53 0 0 0 0 434176 0 0 53 0 53 active+clean 2024-09-18T14:36:52.998666+0000 44'53 46:67 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-11T14:43:33.461488+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 queued for scrub 0 0 2024-09-18T14:45:30.691 INFO:tasks.workunit.client.0.smithi046.stdout:1.5 73 0 0 0 0 598016 0 0 73 0 73 active+clean 2024-09-18T14:36:48.872088+0000 44'73 46:94 [4,2,5] 4 [4,2,5] 4 0'0 2024-09-11T14:43:33.930364+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 queued for scrub 0 0 2024-09-18T14:45:30.691 INFO:tasks.workunit.client.0.smithi046.stdout:2.6 64 0 0 0 0 524288 0 0 64 0 64 active+clean 2024-09-18T14:36:50.618738+0000 44'64 46:82 [1,0,4] 1 [1,0,4] 1 0'0 2024-09-11T14:43:34.430687+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:09:48.734576+0000 0 0 2024-09-18T14:45:30.691 INFO:tasks.workunit.client.0.smithi046.stdout:3.6 64 0 0 0 0 524288 0 0 64 0 64 active+clean 2024-09-18T14:36:52.998021+0000 44'64 46:79 [0,1,4] 0 [0,1,4] 0 0'0 2024-09-11T14:43:34.909707+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 queued for scrub 0 0 2024-09-18T14:45:30.691 INFO:tasks.workunit.client.0.smithi046.stdout:1.4 51 0 0 0 0 417792 0 0 51 0 51 active+clean 2024-09-18T14:36:48.662483+0000 44'51 46:71 [3,0,1] 3 [3,0,1] 3 0'0 2024-09-11T14:43:35.407291+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:13:16.097178+0000 0 0 2024-09-18T14:45:30.691 INFO:tasks.workunit.client.0.smithi046.stdout:2.7 53 0 0 0 0 434176 0 0 53 0 53 active+clean 2024-09-18T14:36:50.937073+0000 44'53 46:69 [3,0,2] 3 [3,0,2] 3 0'0 2024-09-11T14:43:35.879954+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:38:47.644254+0000 0 0 2024-09-18T14:45:30.691 INFO:tasks.workunit.client.0.smithi046.stdout:3.1 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2024-09-18T14:36:52.954138+0000 44'67 46:81 [4,3,5] 4 [4,3,5] 4 0'0 2024-09-11T14:43:36.441821+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 queued for scrub 0 0 2024-09-18T14:45:30.691 INFO:tasks.workunit.client.0.smithi046.stdout:1.3 54 0 0 0 0 442368 0 0 54 0 54 active+clean 2024-09-18T14:36:48.871361+0000 44'54 46:75 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-11T14:43:36.928924+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 queued for scrub 0 0 2024-09-18T14:45:30.691 INFO:tasks.workunit.client.0.smithi046.stdout:2.0 72 0 0 0 0 589824 0 0 72 0 72 active+clean 2024-09-18T14:36:50.621219+0000 44'72 46:88 [3,5,1] 3 [3,5,1] 3 0'0 2024-09-11T14:43:37.417204+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-20T02:06:16.586258+0000 0 0 2024-09-18T14:45:30.692 INFO:tasks.workunit.client.0.smithi046.stdout:3.0 72 0 0 0 0 589824 0 0 72 0 72 active+clean 2024-09-18T14:36:52.956275+0000 44'72 46:86 [1,2,4] 1 [1,2,4] 1 0'0 2024-09-11T14:43:37.894982+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:31:16.804864+0000 0 0 2024-09-18T14:45:30.692 INFO:tasks.workunit.client.0.smithi046.stdout:1.2 55 0 0 0 0 450560 0 0 55 0 55 active+clean 2024-09-18T14:36:48.699754+0000 44'55 46:79 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-11T14:43:38.436840+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 queued for scrub 0 0 2024-09-18T14:45:30.692 INFO:tasks.workunit.client.0.smithi046.stdout:2.1 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2024-09-18T14:36:51.681450+0000 44'67 46:86 [2,5,0] 2 [2,5,0] 2 0'0 2024-09-11T14:43:38.913415+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:23:12.513074+0000 0 0 2024-09-18T14:45:30.692 INFO:tasks.workunit.client.0.smithi046.stdout:3.e 69 0 0 0 0 565248 0 0 69 0 69 active+clean 2024-09-18T14:36:52.918261+0000 44'69 46:83 [2,4,1] 2 [2,4,1] 2 0'0 2024-09-11T14:43:39.401311+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:20:54.223154+0000 0 0 2024-09-18T14:45:30.692 INFO:tasks.workunit.client.0.smithi046.stdout:2.f 62 0 0 0 0 507904 0 0 62 0 62 active+clean 2024-09-18T14:36:50.617118+0000 44'62 46:82 [0,5,4] 0 [0,5,4] 0 0'0 2024-09-11T14:43:39.878926+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 queued for scrub 0 0 2024-09-18T14:45:30.692 INFO:tasks.workunit.client.0.smithi046.stdout:1.c 60 0 0 0 0 491520 0 0 60 0 60 active+clean 2024-09-18T14:36:48.593385+0000 44'60 46:80 [3,0,4] 3 [3,0,4] 3 0'0 2024-09-11T14:43:40.347430+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T23:04:03.428915+0000 0 0 2024-09-18T14:45:30.692 INFO:tasks.workunit.client.0.smithi046.stdout:3.3 54 0 0 0 0 442368 0 0 54 0 54 active+clean 2024-09-18T14:36:52.998753+0000 44'54 46:68 [4,0,5] 4 [4,0,5] 4 0'0 2024-09-11T14:43:40.831017+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 queued for scrub 0 0 2024-09-18T14:45:30.692 INFO:tasks.workunit.client.0.smithi046.stdout:1.1 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2024-09-18T14:36:48.697833+0000 44'67 46:87 [5,0,4] 5 [5,0,4] 5 0'0 2024-09-11T14:43:41.345867+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:42:04.100999+0000 0 0 2024-09-18T14:45:30.692 INFO:tasks.workunit.client.0.smithi046.stdout:2.2 55 0 0 0 0 450560 0 0 55 0 55 active+clean 2024-09-18T14:36:50.622984+0000 44'55 46:71 [5,1,0] 5 [5,1,0] 5 0'0 2024-09-11T14:43:41.813881+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:28:55.204743+0000 0 0 2024-09-18T14:45:30.692 INFO:tasks.workunit.client.0.smithi046.stdout:3.2 55 0 0 0 0 450560 0 0 55 0 55 active+clean 2024-09-18T14:36:52.997928+0000 44'55 46:68 [3,5,0] 3 [3,5,0] 3 0'0 2024-09-11T14:43:42.300039+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:52:49.736896+0000 0 0 2024-09-18T14:45:30.692 INFO:tasks.workunit.client.0.smithi046.stdout:1.0 72 0 0 0 0 589824 0 0 72 0 72 active+clean 2024-09-18T14:36:49.007560+0000 44'72 46:93 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-11T14:43:42.785839+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T22:36:11.932207+0000 0 0 2024-09-18T14:45:30.692 INFO:tasks.workunit.client.0.smithi046.stdout:2.3 54 0 0 0 0 442368 0 0 54 0 54 active+clean 2024-09-18T14:36:50.624348+0000 44'54 46:70 [5,4,1] 5 [5,4,1] 5 0'0 2024-09-11T14:43:43.262049+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-20T02:23:11.701332+0000 0 0 2024-09-18T14:45:30.693 INFO:tasks.workunit.client.0.smithi046.stdout:3.f 62 0 0 0 0 507904 0 0 62 0 62 active+clean 2024-09-18T14:36:52.917739+0000 44'62 46:77 [0,1,4] 0 [0,1,4] 0 0'0 2024-09-11T14:43:43.739831+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 queued for scrub 0 0 2024-09-18T14:45:30.693 INFO:tasks.workunit.client.0.smithi046.stdout:2.e 69 0 0 0 0 565248 0 0 69 0 69 active+clean 2024-09-18T14:36:51.681573+0000 44'69 46:88 [2,3,5] 2 [2,3,5] 2 0'0 2024-09-11T14:43:44.209216+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:46:51.645906+0000 0 0 2024-09-18T14:45:30.693 INFO:tasks.workunit.client.0.smithi046.stdout:1.d 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2024-09-18T14:36:48.081685+0000 44'67 46:90 [2,4,1] 2 [2,4,1] 2 0'0 2024-09-11T14:43:44.725794+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:20:04.204830+0000 0 0 2024-09-18T14:45:30.693 INFO:tasks.workunit.client.0.smithi046.stdout:3.c 60 0 0 0 0 491520 0 0 60 0 60 active+clean 2024-09-18T14:36:52.954220+0000 44'60 46:73 [5,3,1] 5 [5,3,1] 5 0'0 2024-09-11T14:43:45.208366+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T23:05:42.924123+0000 0 0 2024-09-18T14:45:30.693 INFO:tasks.workunit.client.0.smithi046.stdout:2.d 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2024-09-18T14:36:50.623874+0000 44'67 46:83 [5,1,3] 5 [5,1,3] 5 0'0 2024-09-11T14:43:45.730754+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:03:59.303765+0000 0 0 2024-09-18T14:45:30.693 INFO:tasks.workunit.client.0.smithi046.stdout:1.e 69 0 0 0 0 565248 0 0 69 0 69 active+clean 2024-09-18T14:36:49.007425+0000 44'69 46:90 [1,4,2] 1 [1,4,2] 1 0'0 2024-09-11T14:43:46.210549+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:54:04.420111+0000 0 0 2024-09-18T14:45:30.693 INFO:tasks.workunit.client.0.smithi046.stdout:3.d 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2024-09-18T14:36:52.998476+0000 44'67 46:80 [3,1,0] 3 [3,1,0] 3 0'0 2024-09-11T14:43:46.729037+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:16:20.566519+0000 0 0 2024-09-18T14:45:30.693 INFO:tasks.workunit.client.0.smithi046.stdout:1.f 62 0 0 0 0 507904 0 0 62 0 62 active+clean 2024-09-18T14:36:48.593494+0000 44'62 45:81 [3,0,4] 3 [3,0,4] 3 0'0 2024-09-18T14:36:46.926781+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:25:09.813452+0000 0 0 2024-09-18T14:45:30.693 INFO:tasks.workunit.client.0.smithi046.stdout:2.c 60 0 0 0 0 491520 0 0 60 0 60 active+clean 2024-09-18T14:36:51.681038+0000 44'60 46:79 [2,0,5] 2 [2,0,5] 2 0'0 2024-09-11T14:43:47.739402+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:06:14.605209+0000 0 0 2024-09-18T14:45:30.693 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-09-18T14:45:30.693 INFO:tasks.workunit.client.0.smithi046.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:45:30.694 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:30.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:107: TEST_recover_unexpected: ceph pg dump pgs 2024-09-18T14:45:30.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:107: TEST_recover_unexpected: grep +scrubbing 2024-09-18T14:45:31.011 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:31.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:111: TEST_recover_unexpected: sleep 1 2024-09-18T14:45:32.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:104: TEST_recover_unexpected: for i in $(seq 0 300) 2024-09-18T14:45:32.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:106: TEST_recover_unexpected: ceph pg dump pgs 2024-09-18T14:45:32.335 INFO:tasks.workunit.client.0.smithi046.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:45:32.335 INFO:tasks.workunit.client.0.smithi046.stdout:3.9 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2024-09-18T14:36:52.956816+0000 44'61 46:75 [4,2,5] 4 [4,2,5] 4 0'0 2024-09-11T14:43:24.774712+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 queued for scrub 0 0 2024-09-18T14:45:32.335 INFO:tasks.workunit.client.0.smithi046.stdout:1.b 65 0 0 0 0 532480 0 0 65 0 65 active+clean 2024-09-18T14:36:49.007446+0000 44'65 46:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-11T14:43:25.259507+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:46:34.190554+0000 0 0 2024-09-18T14:45:32.335 INFO:tasks.workunit.client.0.smithi046.stdout:2.8 66 0 0 0 0 540672 0 0 66 0 66 active+clean 2024-09-18T14:36:50.939099+0000 44'66 46:86 [0,2,1] 0 [0,2,1] 0 0'0 2024-09-11T14:43:25.731609+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 queued for scrub 0 0 2024-09-18T14:45:32.335 INFO:tasks.workunit.client.0.smithi046.stdout:3.8 66 0 0 0 0 540672 0 0 66 0 66 active+clean 2024-09-18T14:36:52.955028+0000 44'66 46:80 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-11T14:43:26.201260+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 queued for scrub 0 0 2024-09-18T14:45:32.335 INFO:tasks.workunit.client.0.smithi046.stdout:1.a 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2024-09-18T14:36:49.008053+0000 44'61 46:82 [1,2,4] 1 [1,2,4] 1 0'0 2024-09-11T14:43:26.694407+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:17:35.387520+0000 0 0 2024-09-18T14:45:32.335 INFO:tasks.workunit.client.0.smithi046.stdout:2.9 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2024-09-18T14:36:50.619265+0000 44'61 46:78 [4,1,0] 4 [4,1,0] 4 0'0 2024-09-11T14:43:27.166635+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 queued for scrub 0 0 2024-09-18T14:45:32.335 INFO:tasks.workunit.client.0.smithi046.stdout:3.b 65 0 0 0 0 532480 0 0 65 0 65 active+clean 2024-09-18T14:36:53.000633+0000 44'65 46:78 [3,0,4] 3 [3,0,4] 3 0'0 2024-09-11T14:43:27.676050+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:01:51.561657+0000 0 0 2024-09-18T14:45:32.335 INFO:tasks.workunit.client.0.smithi046.stdout:1.9 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2024-09-18T14:36:48.700675+0000 44'61 46:82 [1,4,5] 1 [1,4,5] 1 0'0 2024-09-11T14:43:28.151548+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-20T02:01:25.774857+0000 0 0 2024-09-18T14:45:32.336 INFO:tasks.workunit.client.0.smithi046.stdout:2.a 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2024-09-18T14:36:50.479196+0000 44'61 46:79 [1,5,0] 1 [1,5,0] 1 0'0 2024-09-11T14:43:28.665056+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:41:50.153043+0000 0 0 2024-09-18T14:45:32.336 INFO:tasks.workunit.client.0.smithi046.stdout:3.a 61 0 0 0 0 499712 0 0 61 0 61 active+clean 2024-09-18T14:36:52.955446+0000 44'61 46:74 [3,1,5] 3 [3,1,5] 3 0'0 2024-09-11T14:43:29.144417+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:06:01.652456+0000 0 0 2024-09-18T14:45:32.336 INFO:tasks.workunit.client.0.smithi046.stdout:1.8 66 0 0 0 0 540672 0 0 66 0 66 active+clean 2024-09-18T14:36:48.698901+0000 44'66 46:86 [5,4,3] 5 [5,4,3] 5 0'0 2024-09-11T14:43:29.615673+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:25:38.543673+0000 0 0 2024-09-18T14:45:32.336 INFO:tasks.workunit.client.0.smithi046.stdout:2.b 65 0 0 0 0 532480 0 0 65 0 65 active+clean 2024-09-18T14:36:50.937469+0000 44'65 46:81 [3,2,5] 3 [3,2,5] 3 0'0 2024-09-11T14:43:30.097882+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:31:57.118081+0000 0 0 2024-09-18T14:45:32.336 INFO:tasks.workunit.client.0.smithi046.stdout:3.5 73 0 0 0 0 598016 0 0 73 0 73 active+clean 2024-09-18T14:36:52.953974+0000 44'73 46:86 [5,3,4] 5 [5,3,4] 5 0'0 2024-09-11T14:43:30.580906+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:46:48.338536+0000 0 0 2024-09-18T14:45:32.336 INFO:tasks.workunit.client.0.smithi046.stdout:1.7 53 0 0 0 0 434176 0 0 53 0 53 active+clean 2024-09-18T14:36:48.661525+0000 44'53 46:75 [1,3,0] 1 [1,3,0] 1 0'0 2024-09-11T14:43:31.061428+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:49:40.680653+0000 0 0 2024-09-18T14:45:32.336 INFO:tasks.workunit.client.0.smithi046.stdout:2.4 51 0 0 0 0 417792 0 0 51 0 51 active+clean 2024-09-18T14:36:50.937825+0000 44'51 46:69 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-11T14:43:31.535083+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:59:06.262828+0000 0 0 2024-09-18T14:45:32.336 INFO:tasks.workunit.client.0.smithi046.stdout:3.4 51 0 0 0 0 417792 0 0 51 0 51 active+clean 2024-09-18T14:36:52.954243+0000 44'51 46:65 [1,2,4] 1 [1,2,4] 1 0'0 2024-09-11T14:43:32.019342+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:28:24.899070+0000 0 0 2024-09-18T14:45:32.336 INFO:tasks.workunit.client.0.smithi046.stdout:1.6 64 0 0 0 0 524288 0 0 64 0 64 active+clean 2024-09-18T14:36:48.871192+0000 44'64 46:84 [3,4,2] 3 [3,4,2] 3 0'0 2024-09-11T14:43:32.498437+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T22:48:20.771546+0000 0 0 2024-09-18T14:45:32.336 INFO:tasks.workunit.client.0.smithi046.stdout:2.5 73 0 0 0 0 598016 0 0 73 0 73 active+clean 2024-09-18T14:36:50.618142+0000 44'73 46:89 [3,4,0] 3 [3,4,0] 3 0'0 2024-09-11T14:43:32.990299+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:04:30.830160+0000 0 0 2024-09-18T14:45:32.337 INFO:tasks.workunit.client.0.smithi046.stdout:3.7 53 0 0 0 0 434176 0 0 53 0 53 active+clean 2024-09-18T14:36:52.998666+0000 44'53 46:67 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-11T14:43:33.461488+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 queued for scrub 0 0 2024-09-18T14:45:32.337 INFO:tasks.workunit.client.0.smithi046.stdout:1.5 73 0 0 0 0 598016 0 0 73 0 73 active+clean 2024-09-18T14:36:48.872088+0000 44'73 46:94 [4,2,5] 4 [4,2,5] 4 0'0 2024-09-11T14:43:33.930364+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 queued for scrub 0 0 2024-09-18T14:45:32.337 INFO:tasks.workunit.client.0.smithi046.stdout:2.6 64 0 0 0 0 524288 0 0 64 0 64 active+clean 2024-09-18T14:36:50.618738+0000 44'64 46:82 [1,0,4] 1 [1,0,4] 1 0'0 2024-09-11T14:43:34.430687+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:09:48.734576+0000 0 0 2024-09-18T14:45:32.337 INFO:tasks.workunit.client.0.smithi046.stdout:3.6 64 0 0 0 0 524288 0 0 64 0 64 active+clean 2024-09-18T14:36:52.998021+0000 44'64 46:79 [0,1,4] 0 [0,1,4] 0 0'0 2024-09-11T14:43:34.909707+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 queued for scrub 0 0 2024-09-18T14:45:32.337 INFO:tasks.workunit.client.0.smithi046.stdout:1.4 51 0 0 0 0 417792 0 0 51 0 51 active+clean 2024-09-18T14:36:48.662483+0000 44'51 46:71 [3,0,1] 3 [3,0,1] 3 0'0 2024-09-11T14:43:35.407291+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:13:16.097178+0000 0 0 2024-09-18T14:45:32.337 INFO:tasks.workunit.client.0.smithi046.stdout:2.7 53 0 0 0 0 434176 0 0 53 0 53 active+clean 2024-09-18T14:36:50.937073+0000 44'53 46:69 [3,0,2] 3 [3,0,2] 3 0'0 2024-09-11T14:43:35.879954+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:38:47.644254+0000 0 0 2024-09-18T14:45:32.337 INFO:tasks.workunit.client.0.smithi046.stdout:3.1 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2024-09-18T14:36:52.954138+0000 44'67 46:81 [4,3,5] 4 [4,3,5] 4 0'0 2024-09-11T14:43:36.441821+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 queued for scrub 0 0 2024-09-18T14:45:32.337 INFO:tasks.workunit.client.0.smithi046.stdout:1.3 54 0 0 0 0 442368 0 0 54 0 54 active+clean 2024-09-18T14:36:48.871361+0000 44'54 46:75 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-11T14:43:36.928924+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 queued for scrub 0 0 2024-09-18T14:45:32.337 INFO:tasks.workunit.client.0.smithi046.stdout:2.0 72 0 0 0 0 589824 0 0 72 0 72 active+clean 2024-09-18T14:36:50.621219+0000 44'72 46:88 [3,5,1] 3 [3,5,1] 3 0'0 2024-09-11T14:43:37.417204+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-20T02:06:16.586258+0000 0 0 2024-09-18T14:45:32.337 INFO:tasks.workunit.client.0.smithi046.stdout:3.0 72 0 0 0 0 589824 0 0 72 0 72 active+clean 2024-09-18T14:36:52.956275+0000 44'72 46:86 [1,2,4] 1 [1,2,4] 1 0'0 2024-09-11T14:43:37.894982+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:31:16.804864+0000 0 0 2024-09-18T14:45:32.337 INFO:tasks.workunit.client.0.smithi046.stdout:1.2 55 0 0 0 0 450560 0 0 55 0 55 active+clean 2024-09-18T14:36:48.699754+0000 44'55 46:79 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-11T14:43:38.436840+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 queued for scrub 0 0 2024-09-18T14:45:32.337 INFO:tasks.workunit.client.0.smithi046.stdout:2.1 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2024-09-18T14:36:51.681450+0000 44'67 46:86 [2,5,0] 2 [2,5,0] 2 0'0 2024-09-11T14:43:38.913415+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:23:12.513074+0000 0 0 2024-09-18T14:45:32.337 INFO:tasks.workunit.client.0.smithi046.stdout:3.e 69 0 0 0 0 565248 0 0 69 0 69 active+clean 2024-09-18T14:36:52.918261+0000 44'69 46:83 [2,4,1] 2 [2,4,1] 2 0'0 2024-09-11T14:43:39.401311+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:20:54.223154+0000 0 0 2024-09-18T14:45:32.338 INFO:tasks.workunit.client.0.smithi046.stdout:2.f 62 0 0 0 0 507904 0 0 62 0 62 active+clean 2024-09-18T14:36:50.617118+0000 44'62 46:82 [0,5,4] 0 [0,5,4] 0 0'0 2024-09-11T14:43:39.878926+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 queued for scrub 0 0 2024-09-18T14:45:32.338 INFO:tasks.workunit.client.0.smithi046.stdout:1.c 60 0 0 0 0 491520 0 0 60 0 60 active+clean 2024-09-18T14:36:48.593385+0000 44'60 46:80 [3,0,4] 3 [3,0,4] 3 0'0 2024-09-11T14:43:40.347430+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T23:04:03.428915+0000 0 0 2024-09-18T14:45:32.338 INFO:tasks.workunit.client.0.smithi046.stdout:3.3 54 0 0 0 0 442368 0 0 54 0 54 active+clean 2024-09-18T14:36:52.998753+0000 44'54 46:68 [4,0,5] 4 [4,0,5] 4 0'0 2024-09-11T14:43:40.831017+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 queued for scrub 0 0 2024-09-18T14:45:32.338 INFO:tasks.workunit.client.0.smithi046.stdout:1.1 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2024-09-18T14:36:48.697833+0000 44'67 46:87 [5,0,4] 5 [5,0,4] 5 0'0 2024-09-11T14:43:41.345867+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:42:04.100999+0000 0 0 2024-09-18T14:45:32.338 INFO:tasks.workunit.client.0.smithi046.stdout:2.2 55 0 0 0 0 450560 0 0 55 0 55 active+clean 2024-09-18T14:36:50.622984+0000 44'55 46:71 [5,1,0] 5 [5,1,0] 5 0'0 2024-09-11T14:43:41.813881+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:28:55.204743+0000 0 0 2024-09-18T14:45:32.338 INFO:tasks.workunit.client.0.smithi046.stdout:3.2 55 0 0 0 0 450560 0 0 55 0 55 active+clean 2024-09-18T14:36:52.997928+0000 44'55 46:68 [3,5,0] 3 [3,5,0] 3 0'0 2024-09-11T14:43:42.300039+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:52:49.736896+0000 0 0 2024-09-18T14:45:32.338 INFO:tasks.workunit.client.0.smithi046.stdout:1.0 72 0 0 0 0 589824 0 0 72 0 72 active+clean 2024-09-18T14:36:49.007560+0000 44'72 46:93 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-11T14:43:42.785839+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T22:36:11.932207+0000 0 0 2024-09-18T14:45:32.338 INFO:tasks.workunit.client.0.smithi046.stdout:2.3 54 0 0 0 0 442368 0 0 54 0 54 active+clean 2024-09-18T14:36:50.624348+0000 44'54 46:70 [5,4,1] 5 [5,4,1] 5 0'0 2024-09-11T14:43:43.262049+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-20T02:23:11.701332+0000 0 0 2024-09-18T14:45:32.338 INFO:tasks.workunit.client.0.smithi046.stdout:3.f 62 0 0 0 0 507904 0 0 62 0 62 active+clean 2024-09-18T14:36:52.917739+0000 44'62 46:77 [0,1,4] 0 [0,1,4] 0 0'0 2024-09-11T14:43:43.739831+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 queued for scrub 0 0 2024-09-18T14:45:32.338 INFO:tasks.workunit.client.0.smithi046.stdout:2.e 69 0 0 0 0 565248 0 0 69 0 69 active+clean 2024-09-18T14:36:51.681573+0000 44'69 46:88 [2,3,5] 2 [2,3,5] 2 0'0 2024-09-11T14:43:44.209216+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:46:51.645906+0000 0 0 2024-09-18T14:45:32.338 INFO:tasks.workunit.client.0.smithi046.stdout:1.d 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2024-09-18T14:36:48.081685+0000 44'67 46:90 [2,4,1] 2 [2,4,1] 2 0'0 2024-09-11T14:43:44.725794+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:20:04.204830+0000 0 0 2024-09-18T14:45:32.338 INFO:tasks.workunit.client.0.smithi046.stdout:3.c 60 0 0 0 0 491520 0 0 60 0 60 active+clean 2024-09-18T14:36:52.954220+0000 44'60 46:73 [5,3,1] 5 [5,3,1] 5 0'0 2024-09-11T14:43:45.208366+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-19T23:05:42.924123+0000 0 0 2024-09-18T14:45:32.338 INFO:tasks.workunit.client.0.smithi046.stdout:2.d 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2024-09-18T14:36:50.623874+0000 44'67 46:83 [5,1,3] 5 [5,1,3] 5 0'0 2024-09-11T14:43:45.730754+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:03:59.303765+0000 0 0 2024-09-18T14:45:32.338 INFO:tasks.workunit.client.0.smithi046.stdout:1.e 69 0 0 0 0 565248 0 0 69 0 69 active+clean 2024-09-18T14:36:49.007425+0000 44'69 46:90 [1,4,2] 1 [1,4,2] 1 0'0 2024-09-11T14:43:46.210549+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:54:04.420111+0000 0 0 2024-09-18T14:45:32.339 INFO:tasks.workunit.client.0.smithi046.stdout:3.d 67 0 0 0 0 548864 0 0 67 0 67 active+clean 2024-09-18T14:36:52.998476+0000 44'67 46:80 [3,1,0] 3 [3,1,0] 3 0'0 2024-09-11T14:43:46.729037+0000 0'0 2024-09-18T14:36:51.279535+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:16:20.566519+0000 0 0 2024-09-18T14:45:32.339 INFO:tasks.workunit.client.0.smithi046.stdout:1.f 62 0 0 0 0 507904 0 0 62 0 62 active+clean 2024-09-18T14:36:48.593494+0000 44'62 45:81 [3,0,4] 3 [3,0,4] 3 0'0 2024-09-18T14:36:46.926781+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:25:09.813452+0000 0 0 2024-09-18T14:45:32.339 INFO:tasks.workunit.client.0.smithi046.stdout:2.c 60 0 0 0 0 491520 0 0 60 0 60 active+clean 2024-09-18T14:36:51.681038+0000 44'60 46:79 [2,0,5] 2 [2,0,5] 2 0'0 2024-09-11T14:43:47.739402+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:06:14.605209+0000 0 0 2024-09-18T14:45:32.339 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-09-18T14:45:32.339 INFO:tasks.workunit.client.0.smithi046.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:45:32.339 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:32.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:107: TEST_recover_unexpected: ceph pg dump pgs 2024-09-18T14:45:32.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:107: TEST_recover_unexpected: grep +scrubbing 2024-09-18T14:45:32.661 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:45:32.670 INFO:tasks.workunit.client.0.smithi046.stdout:1.8 66 0 0 0 0 540672 0 0 66 0 66 active+clean+scrubbing 2024-09-18T14:45:31.181092+0000 44'66 47:91 [5,4,3] 5 [5,4,3] 5 0'0 2024-09-11T14:43:29.615673+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 scrubbing for 0s 0 0 2024-09-18T14:45:32.670 INFO:tasks.workunit.client.0.smithi046.stdout:1.c 60 0 0 0 0 491520 0 0 60 0 60 active+clean+scrubbing 2024-09-18T14:45:29.228793+0000 44'60 47:86 [3,0,4] 3 [3,0,4] 3 0'0 2024-09-11T14:43:40.347430+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 scrubbing for 2s 5 0 2024-09-18T14:45:32.670 INFO:tasks.workunit.client.0.smithi046.stdout:2.d 67 0 0 0 0 548864 0 0 67 0 67 active+clean+scrubbing 2024-09-18T14:45:30.162785+0000 44'67 47:88 [5,1,3] 5 [5,1,3] 5 0'0 2024-09-11T14:43:45.730754+0000 0'0 2024-09-18T14:36:48.785595+0000 0 0 scrubbing for 0s 0 0 2024-09-18T14:45:32.670 INFO:tasks.workunit.client.0.smithi046.stdout:1.e 69 0 0 0 0 565248 0 0 69 0 69 active+clean+scrubbing 2024-09-18T14:45:29.582730+0000 44'69 47:96 [1,4,2] 1 [1,4,2] 1 0'0 2024-09-11T14:43:46.210549+0000 0'0 2024-09-18T14:36:46.926781+0000 0 0 scrubbing for 2s 5 0 2024-09-18T14:45:32.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:108: TEST_recover_unexpected: ok=true 2024-09-18T14:45:32.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:109: TEST_recover_unexpected: break 2024-09-18T14:45:32.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:113: TEST_recover_unexpected: test true = false 2024-09-18T14:45:32.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:118: TEST_recover_unexpected: local total=0 2024-09-18T14:45:32.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:119: TEST_recover_unexpected: local zerocount=0 2024-09-18T14:45:32.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: local maxzerocount=3 2024-09-18T14:45:32.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:45:32.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:45:32.673 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:45:32.675 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:45:32.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:32.677 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:45:32.677 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:45:32.677 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:45:32.678 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:32.678 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:32.678 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:32.678 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:45:32.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:32.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:45:32.761 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:32.761 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:45:32.761 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 4, 2024-09-18T14:45:32.761 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:32.761 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:32.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:32.771 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:45:32.772 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:45:32.772 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:45:32.772 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:32.772 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:32.772 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:32.772 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:45:32.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:32.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:45:32.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:45:32.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:45:32.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 4 2024-09-18T14:45:32.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=4 2024-09-18T14:45:32.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:32.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:45:32.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:45:32.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:45:32.870 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:32.870 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:32.870 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:32.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:45:32.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:32.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:45:32.958 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:32.958 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:45:32.958 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:45:32.958 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:32.958 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:32.967 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:32.968 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:45:32.968 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:45:32.968 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:45:32.969 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:32.969 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:32.969 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:32.969 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:45:32.969 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:32.969 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:45:33.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:45:33.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:45:33.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 4 + 2 2024-09-18T14:45:33.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=6 2024-09-18T14:45:33.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:33.068 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:45:33.069 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:45:33.069 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:45:33.069 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:33.069 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:33.070 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:33.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:45:33.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:33.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:45:33.158 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:33.158 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:45:33.158 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:45:33.158 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:33.158 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:33.168 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:33.168 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:45:33.168 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:45:33.168 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:45:33.172 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:33.172 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:33.172 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:33.172 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:45:33.172 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:33.172 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:45:33.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:45:33.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:45:33.264 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 6 + 3 2024-09-18T14:45:33.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=9 2024-09-18T14:45:33.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:33.265 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:45:33.265 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:45:33.265 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:45:33.266 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:33.266 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:33.266 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:33.266 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:45:33.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:33.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:45:33.347 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:33.347 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:45:33.347 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:45:33.347 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:33.347 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:33.356 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:33.357 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:45:33.357 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:45:33.357 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:45:33.358 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:33.358 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:33.358 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:33.358 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:45:33.359 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:33.359 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:45:33.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:45:33.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:45:33.455 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 9 + 4 2024-09-18T14:45:33.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=13 2024-09-18T14:45:33.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:33.457 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:45:33.457 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:45:33.457 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:45:33.458 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:33.458 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:33.458 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:33.458 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:45:33.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:33.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:45:33.545 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:33.545 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:45:33.545 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:45:33.545 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:33.545 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:33.554 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:33.555 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:45:33.555 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:45:33.555 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:45:33.556 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:33.556 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:33.556 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:33.556 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:45:33.556 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:33.556 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:45:33.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:45:33.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:45:33.651 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 13 + 4 2024-09-18T14:45:33.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=17 2024-09-18T14:45:33.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:33.654 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:45:33.654 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:45:33.654 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:45:33.655 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:33.655 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:33.655 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:33.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:45:33.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:33.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:45:33.740 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:33.740 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:45:33.741 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:45:33.741 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:33.741 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:33.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:33.750 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:45:33.750 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:45:33.750 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:45:33.751 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:33.751 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:33.751 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:33.752 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:45:33.753 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:33.753 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:45:33.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:45:33.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:45:33.849 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 17 + 4 2024-09-18T14:45:33.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=21 2024-09-18T14:45:33.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 21 = 0 ']' 2024-09-18T14:45:33.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:45:33.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 0 + 21 2024-09-18T14:45:33.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=21 2024-09-18T14:45:33.854 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:45:33.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:45:37.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:45:37.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:45:37.859 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:45:37.861 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:45:37.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:37.864 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:45:37.864 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:45:37.864 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:45:37.865 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:37.865 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:37.865 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:37.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:45:37.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:37.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:45:37.960 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:37.960 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:45:37.960 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 4, 2024-09-18T14:45:37.960 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:37.960 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:37.969 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:37.970 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:45:37.970 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:45:37.970 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:45:37.971 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:37.971 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:37.971 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:37.972 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:45:37.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:37.973 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:45:38.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:45:38.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:45:38.068 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 4 2024-09-18T14:45:38.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=4 2024-09-18T14:45:38.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:38.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:45:38.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:45:38.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:45:38.071 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:38.071 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:38.072 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:38.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:45:38.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:38.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:45:38.159 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:38.159 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:45:38.159 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:45:38.159 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:38.159 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:38.168 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:38.169 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:45:38.169 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:45:38.169 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:45:38.170 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:38.170 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:38.170 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:38.170 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:45:38.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:38.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:45:38.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:45:38.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:45:38.265 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 4 + 2 2024-09-18T14:45:38.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=6 2024-09-18T14:45:38.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:38.268 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:45:38.268 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:45:38.268 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:45:38.269 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:38.269 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:38.269 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:38.269 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:45:38.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:38.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:45:38.357 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:38.357 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:45:38.357 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:45:38.357 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:38.357 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:38.366 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:38.367 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:45:38.367 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:45:38.367 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:45:38.367 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:38.367 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:38.367 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:38.368 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:45:38.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:38.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:45:38.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:45:38.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:45:38.462 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 6 + 3 2024-09-18T14:45:38.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=9 2024-09-18T14:45:38.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:38.465 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:45:38.465 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:45:38.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:45:38.466 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:38.466 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:38.466 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:38.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:45:38.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:38.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:45:38.552 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:38.553 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:45:38.553 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:45:38.553 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:38.553 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:38.562 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:38.562 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:45:38.562 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:45:38.563 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:45:38.563 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:38.563 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:38.563 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:38.563 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:45:38.564 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:38.564 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:45:38.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:45:38.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:45:38.660 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 9 + 4 2024-09-18T14:45:38.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=13 2024-09-18T14:45:38.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:38.662 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:45:38.662 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:45:38.662 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:45:38.662 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:38.662 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:38.662 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:38.663 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:45:38.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:38.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:45:38.750 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:38.750 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:45:38.750 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:45:38.750 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:38.750 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:38.760 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:38.760 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:45:38.760 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:45:38.760 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:45:38.760 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:38.760 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:38.760 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:38.760 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:45:38.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:38.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:45:38.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:45:38.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:45:38.856 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 13 + 4 2024-09-18T14:45:38.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=17 2024-09-18T14:45:38.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:38.858 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:45:38.858 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:45:38.858 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:45:38.858 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:38.858 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:38.858 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:38.859 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:45:38.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:38.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:45:38.939 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:38.939 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:45:38.939 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:45:38.939 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:38.939 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:38.949 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:38.949 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:45:38.949 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:45:38.950 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:45:38.950 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:38.950 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:38.950 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:38.950 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:45:38.950 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:38.950 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:45:39.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:45:39.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:45:39.043 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 17 + 4 2024-09-18T14:45:39.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=21 2024-09-18T14:45:39.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 21 = 0 ']' 2024-09-18T14:45:39.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:45:39.045 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 21 + 21 2024-09-18T14:45:39.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=42 2024-09-18T14:45:39.047 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:45:39.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:45:43.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:45:43.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:45:43.052 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:45:43.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:45:43.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:43.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:45:43.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:45:43.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:45:43.058 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:43.058 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:43.058 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:43.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:45:43.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:43.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:45:43.152 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:43.152 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:45:43.152 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 4, 2024-09-18T14:45:43.152 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:43.152 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:43.162 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:43.163 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:45:43.163 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:45:43.163 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:45:43.163 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:43.163 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:43.164 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:43.164 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:45:43.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:43.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:45:43.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:45:43.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:45:43.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 4 2024-09-18T14:45:43.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=4 2024-09-18T14:45:43.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:43.265 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:45:43.265 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:45:43.265 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:45:43.265 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:43.265 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:43.266 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:43.266 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:45:43.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:43.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:45:43.348 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:43.348 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:45:43.348 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:45:43.348 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:43.349 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:43.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:43.358 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:45:43.358 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:45:43.359 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:45:43.359 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:43.359 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:43.359 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:43.360 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:45:43.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:43.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:45:43.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:45:43.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:45:43.457 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 4 + 2 2024-09-18T14:45:43.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=6 2024-09-18T14:45:43.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:43.460 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:45:43.460 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:45:43.460 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:45:43.461 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:43.461 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:43.461 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:43.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:45:43.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:43.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:45:43.546 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:43.546 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:45:43.546 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:45:43.546 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:43.546 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:43.556 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:43.557 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:45:43.557 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:45:43.557 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:45:43.557 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:43.558 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:43.558 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:43.558 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:45:43.558 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:43.558 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:45:43.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:45:43.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:45:43.649 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 6 + 3 2024-09-18T14:45:43.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=9 2024-09-18T14:45:43.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:43.652 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:45:43.652 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:45:43.652 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:45:43.653 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:43.653 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:43.653 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:43.653 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:45:43.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:43.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:45:43.739 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:43.739 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:45:43.739 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:45:43.740 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:43.740 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:43.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:43.749 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:45:43.749 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:45:43.750 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:45:43.750 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:43.750 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:43.750 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:43.751 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:45:43.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:43.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:45:43.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:45:43.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:45:43.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 9 + 4 2024-09-18T14:45:43.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=13 2024-09-18T14:45:43.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:43.845 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:45:43.845 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:45:43.845 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:45:43.845 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:43.845 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:43.845 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:43.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:45:43.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:43.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:45:43.932 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:43.932 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:45:43.932 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:45:43.932 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:43.933 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:43.950 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:43.950 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:45:43.950 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:45:43.950 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:45:43.951 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:43.951 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:43.951 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:43.951 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:45:43.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:43.952 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:45:44.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:45:44.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:45:44.046 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 13 + 4 2024-09-18T14:45:44.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=17 2024-09-18T14:45:44.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:44.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:45:44.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:45:44.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:45:44.049 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:44.049 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:44.049 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:44.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:45:44.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:44.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:45:44.135 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:44.135 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:45:44.135 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:45:44.135 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:44.135 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:44.144 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:44.145 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:45:44.145 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:45:44.145 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:45:44.146 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:44.146 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:44.146 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:44.146 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:45:44.147 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:44.147 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:45:44.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:45:44.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:45:44.242 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 17 + 4 2024-09-18T14:45:44.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=21 2024-09-18T14:45:44.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 21 = 0 ']' 2024-09-18T14:45:44.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:45:44.244 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 42 + 21 2024-09-18T14:45:44.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=63 2024-09-18T14:45:44.247 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:45:44.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:45:48.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:45:48.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:45:48.253 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:45:48.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:45:48.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:48.257 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:45:48.257 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:45:48.257 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:45:48.258 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:48.258 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:48.258 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:48.259 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:45:48.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:48.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:45:48.351 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:48.351 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:45:48.352 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 4, 2024-09-18T14:45:48.352 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:48.352 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:48.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:48.362 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:45:48.362 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:45:48.362 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:45:48.362 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:48.363 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:48.363 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:48.363 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:45:48.364 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:48.364 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:45:48.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:45:48.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:45:48.456 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 4 2024-09-18T14:45:48.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=4 2024-09-18T14:45:48.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:48.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:45:48.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:45:48.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:45:48.459 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:48.459 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:48.459 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:48.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:45:48.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:48.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:45:48.547 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:48.547 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:45:48.547 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:45:48.547 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:48.547 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:48.556 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:48.557 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:45:48.557 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:45:48.557 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:45:48.558 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:48.558 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:48.558 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:48.558 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:45:48.558 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:48.559 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:45:48.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:45:48.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:45:48.654 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 4 + 2 2024-09-18T14:45:48.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=6 2024-09-18T14:45:48.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:48.657 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:45:48.657 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:45:48.657 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:45:48.658 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:48.658 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:48.658 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:48.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:45:48.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:48.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:45:48.739 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:48.739 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:45:48.740 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:45:48.740 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:48.740 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:48.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:48.749 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:45:48.749 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:45:48.749 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:45:48.750 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:48.750 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:48.750 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:48.750 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:45:48.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:48.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:45:48.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:45:48.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:45:48.839 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 6 + 3 2024-09-18T14:45:48.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=9 2024-09-18T14:45:48.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:48.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:45:48.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:45:48.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:45:48.842 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:48.842 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:48.842 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:48.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:45:48.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:48.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:45:48.930 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:48.930 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:45:48.930 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:45:48.930 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:48.930 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:48.940 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:48.940 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:45:48.940 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:45:48.940 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:45:48.940 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:48.940 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:48.941 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:48.941 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:45:48.941 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:48.941 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:45:49.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:45:49.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:45:49.039 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 9 + 4 2024-09-18T14:45:49.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=13 2024-09-18T14:45:49.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:49.041 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:45:49.042 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:45:49.042 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:45:49.042 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:49.042 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:49.043 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:49.043 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:45:49.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:49.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:45:49.124 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:49.124 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:45:49.124 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:45:49.124 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:49.125 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:49.134 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:49.134 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:45:49.134 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:45:49.135 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:45:49.135 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:49.135 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:49.135 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:49.136 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:45:49.136 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:49.136 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:45:49.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:45:49.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:45:49.233 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 13 + 4 2024-09-18T14:45:49.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=17 2024-09-18T14:45:49.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:49.235 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:45:49.235 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:45:49.235 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:45:49.236 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:49.236 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:49.236 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:49.236 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:45:49.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:49.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:45:49.317 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:49.317 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:45:49.317 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:45:49.317 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:49.317 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:49.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:49.326 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:45:49.326 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:45:49.326 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:45:49.327 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:49.327 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:49.327 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:49.328 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:45:49.328 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:49.328 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:45:49.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:45:49.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:45:49.427 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 17 + 4 2024-09-18T14:45:49.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=21 2024-09-18T14:45:49.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 21 = 0 ']' 2024-09-18T14:45:49.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:45:49.430 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 63 + 21 2024-09-18T14:45:49.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=84 2024-09-18T14:45:49.432 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:45:49.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:45:53.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:45:53.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:45:53.438 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:45:53.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:45:53.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:53.446 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:45:53.446 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:45:53.446 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:45:53.446 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:53.446 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:53.446 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:53.446 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:45:53.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:53.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:45:53.537 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:53.537 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:45:53.537 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:45:53.537 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:53.537 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:53.546 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:53.546 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:45:53.546 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:45:53.546 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:45:53.547 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:53.547 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:53.547 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:53.547 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:45:53.548 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:53.548 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:45:53.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:45:53.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:45:53.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 2 2024-09-18T14:45:53.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=2 2024-09-18T14:45:53.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:53.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:45:53.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:45:53.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:45:53.647 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:53.647 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:53.647 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:53.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:45:53.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:53.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:45:53.733 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:53.733 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:45:53.733 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:45:53.733 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:53.733 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:53.742 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:53.743 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:45:53.743 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:45:53.743 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:45:53.744 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:53.744 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:53.744 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:53.744 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:45:53.744 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:53.744 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:45:53.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:45:53.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:45:53.836 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 2 + 2 2024-09-18T14:45:53.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=4 2024-09-18T14:45:53.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:53.838 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:45:53.838 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:45:53.838 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:45:53.838 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:53.839 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:53.839 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:53.839 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:45:53.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:53.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:45:53.927 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:53.927 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:45:53.927 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:45:53.927 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:53.927 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:53.936 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:53.937 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:45:53.937 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:45:53.937 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:45:53.937 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:53.938 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:53.938 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:53.938 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:45:53.938 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:53.938 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:45:54.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:45:54.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:45:54.027 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 4 + 4 2024-09-18T14:45:54.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=8 2024-09-18T14:45:54.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:54.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:45:54.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:45:54.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:45:54.031 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:54.031 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:54.031 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:54.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:45:54.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:54.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:45:54.116 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:54.116 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:45:54.117 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:45:54.117 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:54.117 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:54.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:54.126 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:45:54.126 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:45:54.126 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:45:54.127 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:54.127 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:54.127 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:54.127 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:45:54.128 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:54.128 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:45:54.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:45:54.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:45:54.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 8 + 2 2024-09-18T14:45:54.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=10 2024-09-18T14:45:54.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:54.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:45:54.221 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:45:54.221 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:45:54.221 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:54.221 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:54.221 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:54.221 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:45:54.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:54.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:45:54.303 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:54.303 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:45:54.303 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:45:54.304 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:54.304 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:54.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:54.312 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:45:54.312 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:45:54.312 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:45:54.313 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:54.313 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:54.313 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:54.313 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:45:54.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:54.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:45:54.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:45:54.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:45:54.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 10 + 3 2024-09-18T14:45:54.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=13 2024-09-18T14:45:54.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:54.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:45:54.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:45:54.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:45:54.414 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:54.414 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:54.415 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:54.415 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:45:54.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:54.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:45:54.511 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:54.512 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:45:54.512 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:45:54.512 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:54.512 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:54.522 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:54.522 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:45:54.522 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:45:54.522 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:45:54.522 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:54.522 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:54.522 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:54.523 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:45:54.523 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:54.523 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:45:54.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:45:54.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:45:54.617 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 13 + 4 2024-09-18T14:45:54.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=17 2024-09-18T14:45:54.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 17 = 0 ']' 2024-09-18T14:45:54.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:45:54.619 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 84 + 17 2024-09-18T14:45:54.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=101 2024-09-18T14:45:54.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:45:54.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:45:58.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:45:58.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:45:58.627 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:45:58.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:45:58.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:58.631 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:45:58.631 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:45:58.631 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:45:58.632 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:58.632 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:58.632 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:58.633 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:45:58.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:58.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:45:58.723 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:58.723 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:45:58.724 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:45:58.724 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:58.724 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:58.734 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:58.734 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:45:58.734 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:45:58.734 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:45:58.735 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:58.735 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:58.735 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:58.736 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:45:58.736 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:58.736 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:45:58.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:45:58.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:45:58.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 1 2024-09-18T14:45:58.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=1 2024-09-18T14:45:58.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:58.832 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:45:58.832 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:45:58.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:45:58.833 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:58.833 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:58.833 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:58.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:45:58.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:58.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:45:58.917 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:58.918 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:45:58.918 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:45:58.918 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:58.918 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:58.927 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:58.928 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:45:58.928 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:45:58.929 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:45:58.929 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:58.929 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:58.929 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:58.929 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:45:58.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:58.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:45:59.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:45:59.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:45:59.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 1 + 3 2024-09-18T14:45:59.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=4 2024-09-18T14:45:59.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:59.024 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:45:59.024 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:45:59.024 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:45:59.025 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:59.025 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:59.025 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:59.025 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:45:59.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:59.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:45:59.109 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:59.110 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:45:59.110 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:45:59.110 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:59.110 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:59.120 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:59.120 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:45:59.120 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:45:59.120 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:45:59.120 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:59.120 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:59.120 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:59.120 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:45:59.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:59.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:45:59.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:45:59.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:45:59.215 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 4 + 4 2024-09-18T14:45:59.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=8 2024-09-18T14:45:59.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:59.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:45:59.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:45:59.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:45:59.219 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:59.219 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:59.219 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:59.219 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:45:59.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:59.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:45:59.306 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:59.306 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:45:59.306 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:45:59.306 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:59.306 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:59.317 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:59.317 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:45:59.317 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:45:59.317 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:45:59.318 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:59.318 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:59.318 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:59.318 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:45:59.318 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:59.318 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:45:59.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:45:59.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:45:59.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 8 + 2 2024-09-18T14:45:59.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=10 2024-09-18T14:45:59.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:59.409 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:45:59.409 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:45:59.409 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:45:59.410 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:59.410 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:59.410 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:59.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:45:59.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:59.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:45:59.497 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:59.497 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:45:59.497 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:45:59.497 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:59.497 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:59.507 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:59.507 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:45:59.507 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:45:59.508 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:45:59.508 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:59.508 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:59.508 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:59.508 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:45:59.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:59.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:45:59.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:45:59.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:45:59.602 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 10 + 4 2024-09-18T14:45:59.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=14 2024-09-18T14:45:59.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:45:59.605 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:45:59.606 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:45:59.606 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:45:59.606 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:59.606 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:59.606 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:59.606 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:45:59.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:59.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:45:59.694 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:45:59.694 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:45:59.694 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:45:59.694 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:45:59.694 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:45:59.704 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:45:59.704 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:45:59.704 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:45:59.704 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:45:59.706 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:45:59.707 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:59.707 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:45:59.707 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:45:59.707 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:45:59.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:45:59.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:45:59.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:45:59.804 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 14 + 1 2024-09-18T14:45:59.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=15 2024-09-18T14:45:59.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 15 = 0 ']' 2024-09-18T14:45:59.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:45:59.806 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 101 + 15 2024-09-18T14:45:59.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=116 2024-09-18T14:45:59.807 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:45:59.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:46:03.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:46:03.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:46:03.813 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:46:03.814 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:46:03.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:03.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:46:03.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:46:03.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:46:03.818 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:03.818 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:03.818 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:03.818 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:46:03.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:03.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:46:03.913 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:03.913 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:03.913 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:03.913 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:03.913 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:03.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:03.923 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:46:03.923 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:46:03.923 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:46:03.924 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:03.924 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:03.924 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:03.924 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:46:03.925 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:03.925 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:46:04.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:46:04.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:46:04.020 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 3 2024-09-18T14:46:04.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=3 2024-09-18T14:46:04.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:04.023 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:46:04.023 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:46:04.023 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:46:04.024 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:04.024 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:04.024 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:04.024 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:46:04.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:04.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:46:04.111 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:04.111 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:04.111 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:46:04.111 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:04.111 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:04.120 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:04.121 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:46:04.121 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:46:04.121 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:46:04.121 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:04.121 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:04.122 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:04.122 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:46:04.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:04.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:46:04.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:04.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:04.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 3 + 4 2024-09-18T14:46:04.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=7 2024-09-18T14:46:04.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:04.221 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:46:04.221 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:46:04.221 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:46:04.222 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:04.222 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:04.222 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:04.222 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:46:04.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:04.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:46:04.307 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:04.307 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:46:04.307 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:04.307 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:04.308 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:04.316 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:04.317 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:46:04.317 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:46:04.317 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:46:04.318 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:04.318 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:04.318 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:04.318 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:46:04.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:04.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:46:04.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:04.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:04.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 7 + 4 2024-09-18T14:46:04.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=11 2024-09-18T14:46:04.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:04.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:46:04.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:46:04.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:46:04.415 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:04.415 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:04.415 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:04.415 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:46:04.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:04.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:46:04.501 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:04.502 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:46:04.502 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:46:04.502 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:04.502 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:04.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:04.512 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:46:04.512 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:46:04.512 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:46:04.513 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:04.513 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:04.513 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:04.513 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:46:04.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:04.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:46:04.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:46:04.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:46:04.610 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 11 + 2 2024-09-18T14:46:04.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=13 2024-09-18T14:46:04.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:04.613 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:46:04.613 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:46:04.613 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:46:04.613 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:04.614 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:04.614 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:04.614 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:46:04.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:04.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:46:04.698 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:04.699 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:46:04.699 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 4, 2024-09-18T14:46:04.699 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:04.699 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:04.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:04.709 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:46:04.709 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:46:04.709 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:46:04.709 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:04.710 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:04.710 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:04.710 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:46:04.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:04.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:46:04.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:04.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:04.804 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 13 + 4 2024-09-18T14:46:04.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=17 2024-09-18T14:46:04.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:04.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:46:04.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:46:04.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:46:04.806 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:04.806 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:04.806 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:04.807 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:46:04.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:04.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:46:04.890 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:04.890 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:46:04.890 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:46:04.890 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:04.890 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:04.899 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:04.900 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:46:04.900 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:46:04.900 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:46:04.901 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:04.901 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:04.901 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:04.901 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:46:04.902 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:04.902 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:46:04.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:46:04.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:46:05.000 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 17 + 1 2024-09-18T14:46:05.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=18 2024-09-18T14:46:05.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 18 = 0 ']' 2024-09-18T14:46:05.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:46:05.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 116 + 18 2024-09-18T14:46:05.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=134 2024-09-18T14:46:05.004 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:46:05.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:46:09.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:46:09.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:46:09.010 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:46:09.011 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:46:09.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:09.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:46:09.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:46:09.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:46:09.015 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:09.015 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:09.015 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:09.015 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:46:09.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:09.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:46:09.111 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:09.111 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:09.111 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:09.112 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:09.112 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:09.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:09.122 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:46:09.123 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:46:09.123 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:46:09.123 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:09.123 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:09.123 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:09.124 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:46:09.124 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:09.124 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:46:09.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:46:09.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:46:09.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 3 2024-09-18T14:46:09.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=3 2024-09-18T14:46:09.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:09.219 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:46:09.219 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:46:09.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:46:09.220 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:09.220 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:09.221 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:09.221 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:46:09.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:09.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:46:09.306 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:09.306 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:09.306 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:46:09.306 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:09.306 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:09.316 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:09.316 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:46:09.316 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:46:09.316 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:46:09.317 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:09.317 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:09.318 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:09.318 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:46:09.318 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:09.318 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:46:09.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:09.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:09.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 3 + 4 2024-09-18T14:46:09.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=7 2024-09-18T14:46:09.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:09.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:46:09.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:46:09.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:46:09.414 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:09.414 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:09.414 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:09.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:46:09.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:09.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:46:09.501 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:09.501 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:46:09.501 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:09.501 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:09.501 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:09.510 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:09.510 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:46:09.510 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:46:09.511 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:46:09.511 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:09.511 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:09.511 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:09.512 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:46:09.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:09.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:46:09.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:09.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:09.609 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 7 + 4 2024-09-18T14:46:09.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=11 2024-09-18T14:46:09.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:09.612 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:46:09.612 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:46:09.613 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:46:09.613 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:09.613 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:09.613 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:09.613 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:46:09.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:09.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:46:09.700 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:09.700 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:46:09.700 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:46:09.700 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:09.700 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:09.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:09.709 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:46:09.709 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:46:09.710 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:46:09.710 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:09.710 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:09.711 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:09.711 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:46:09.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:09.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:46:09.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:46:09.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:46:09.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 11 + 2 2024-09-18T14:46:09.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=13 2024-09-18T14:46:09.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:09.809 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:46:09.809 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:46:09.809 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:46:09.809 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:09.809 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:09.809 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:09.810 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:46:09.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:09.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:46:09.894 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:09.894 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:46:09.894 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 4, 2024-09-18T14:46:09.894 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:09.894 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:09.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:09.903 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:46:09.903 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:46:09.903 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:46:09.904 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:09.904 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:09.904 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:09.905 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:46:09.906 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:09.906 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:46:09.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:09.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:09.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 13 + 4 2024-09-18T14:46:09.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=17 2024-09-18T14:46:09.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:09.998 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:46:09.998 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:46:09.998 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:46:09.999 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:09.999 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:09.999 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:09.999 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:46:09.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:10.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:46:10.088 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:10.088 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:46:10.088 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:46:10.088 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:10.088 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:10.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:10.098 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:46:10.098 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:46:10.098 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:46:10.099 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:10.099 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:10.099 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:10.100 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:46:10.100 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:10.101 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:46:10.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:46:10.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:46:10.195 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 17 + 1 2024-09-18T14:46:10.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=18 2024-09-18T14:46:10.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 18 = 0 ']' 2024-09-18T14:46:10.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:46:10.197 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 134 + 18 2024-09-18T14:46:10.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=152 2024-09-18T14:46:10.199 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:46:10.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:46:14.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:46:14.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:46:14.205 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:46:14.206 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:46:14.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:14.209 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:46:14.210 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:46:14.210 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:46:14.211 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:14.211 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:14.211 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:14.211 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:46:14.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:14.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:46:14.302 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:14.302 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:14.303 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:14.303 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:14.303 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:14.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:14.313 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:46:14.313 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:46:14.313 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:46:14.314 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:14.314 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:14.314 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:14.314 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:46:14.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:14.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:46:14.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:46:14.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:46:14.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 3 2024-09-18T14:46:14.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=3 2024-09-18T14:46:14.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:14.412 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:46:14.412 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:46:14.412 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:46:14.412 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:14.412 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:14.412 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:14.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:46:14.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:14.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:46:14.498 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:14.498 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:14.499 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:46:14.499 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:14.499 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:14.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:14.509 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:46:14.509 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:46:14.509 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:46:14.509 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:14.509 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:14.509 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:14.509 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:46:14.510 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:14.510 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:46:14.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:14.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:14.602 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 3 + 4 2024-09-18T14:46:14.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=7 2024-09-18T14:46:14.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:14.605 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:46:14.605 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:46:14.605 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:46:14.606 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:14.606 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:14.606 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:14.606 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:46:14.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:14.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:46:14.692 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:14.693 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:46:14.693 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:14.693 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:14.693 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:14.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:14.702 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:46:14.702 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:46:14.702 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:46:14.703 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:14.703 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:14.703 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:14.703 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:46:14.704 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:14.704 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:46:14.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:14.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:14.799 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 7 + 4 2024-09-18T14:46:14.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=11 2024-09-18T14:46:14.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:14.802 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:46:14.802 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:46:14.802 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:46:14.802 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:14.802 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:14.803 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:14.803 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:46:14.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:14.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:46:14.889 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:14.889 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:46:14.889 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:46:14.889 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:14.889 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:14.902 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:14.903 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:46:14.903 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:46:14.903 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:46:14.903 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:14.904 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:14.904 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:14.904 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:46:14.904 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:14.904 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:46:14.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:46:14.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:46:14.989 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 11 + 2 2024-09-18T14:46:14.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=13 2024-09-18T14:46:14.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:14.991 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:46:14.991 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:46:14.991 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:46:14.992 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:14.992 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:14.992 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:14.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:46:14.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:14.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:46:15.078 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:15.078 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:46:15.078 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 4, 2024-09-18T14:46:15.078 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:15.078 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:15.088 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:15.088 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:46:15.088 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:46:15.088 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:46:15.089 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:15.089 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:15.089 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:15.089 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:46:15.089 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:15.089 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:46:15.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:15.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:15.179 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 13 + 4 2024-09-18T14:46:15.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=17 2024-09-18T14:46:15.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:15.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:46:15.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:46:15.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:46:15.182 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:15.182 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:15.182 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:15.183 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:46:15.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:15.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:46:15.271 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:15.271 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:46:15.271 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:46:15.271 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:15.272 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:15.282 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:15.282 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:46:15.282 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:46:15.282 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:46:15.283 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:15.283 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:15.283 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:15.284 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:46:15.285 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:15.285 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:46:15.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:46:15.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:46:15.380 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 17 + 1 2024-09-18T14:46:15.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=18 2024-09-18T14:46:15.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 18 = 0 ']' 2024-09-18T14:46:15.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:46:15.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 152 + 18 2024-09-18T14:46:15.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=170 2024-09-18T14:46:15.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:46:15.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:46:19.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:46:19.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:46:19.391 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:46:19.392 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:46:19.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:19.395 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:46:19.396 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:46:19.396 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:46:19.396 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:19.396 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:19.396 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:19.397 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:46:19.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:19.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:46:19.489 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:19.489 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:19.489 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:46:19.489 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:19.489 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:19.498 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:19.499 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:46:19.499 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:46:19.499 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:46:19.500 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:19.500 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:19.500 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:19.500 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:46:19.501 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:19.501 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:46:19.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:19.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:19.599 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 4 2024-09-18T14:46:19.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=4 2024-09-18T14:46:19.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:19.602 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:46:19.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:46:19.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:46:19.603 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:19.603 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:19.603 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:19.604 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:46:19.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:19.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:46:19.690 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:19.690 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:19.690 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:46:19.690 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:19.690 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:19.699 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:19.700 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:46:19.700 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:46:19.700 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:46:19.700 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:19.701 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:19.701 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:19.701 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:46:19.702 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:19.702 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:46:19.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:19.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:19.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 4 + 4 2024-09-18T14:46:19.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=8 2024-09-18T14:46:19.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:19.798 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:46:19.798 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:46:19.798 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:46:19.799 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:19.799 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:19.799 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:19.799 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:46:19.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:19.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:46:19.880 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:19.880 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:46:19.880 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:19.880 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:19.880 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:19.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:19.890 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:46:19.890 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:46:19.890 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:46:19.890 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:19.891 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:19.891 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:19.891 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:46:19.892 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:19.892 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:46:19.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:19.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:19.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 8 + 4 2024-09-18T14:46:19.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=12 2024-09-18T14:46:19.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:19.988 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:46:19.988 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:46:19.988 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:46:19.988 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:19.989 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:19.989 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:19.989 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:46:19.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:19.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:46:20.074 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:20.074 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 3, 2024-09-18T14:46:20.074 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:46:20.074 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:20.074 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:20.084 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:20.084 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:46:20.084 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:46:20.084 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:46:20.085 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:20.085 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:20.085 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:20.086 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:46:20.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:20.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:46:20.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:46:20.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:46:20.177 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 12 + 3 2024-09-18T14:46:20.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=15 2024-09-18T14:46:20.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:20.180 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:46:20.180 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:46:20.180 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:46:20.180 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:20.181 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:20.181 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:20.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:46:20.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:20.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:46:20.268 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:20.269 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:46:20.269 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 4, 2024-09-18T14:46:20.269 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:20.269 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:20.278 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:20.278 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:46:20.279 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:46:20.279 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:46:20.279 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:20.279 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:20.279 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:20.279 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:46:20.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:20.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:46:20.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:20.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:20.374 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 15 + 4 2024-09-18T14:46:20.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=19 2024-09-18T14:46:20.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:20.377 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:46:20.377 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:46:20.377 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:46:20.378 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:20.378 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:20.378 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:20.378 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:46:20.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:20.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:46:20.462 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:20.462 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:46:20.462 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:20.462 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:20.463 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:20.471 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:20.472 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:46:20.472 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:46:20.472 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:46:20.473 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:20.473 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:20.473 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:20.473 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:46:20.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:20.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:46:20.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:46:20.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:46:20.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 19 + 2 2024-09-18T14:46:20.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=21 2024-09-18T14:46:20.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 21 = 0 ']' 2024-09-18T14:46:20.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:46:20.570 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 170 + 21 2024-09-18T14:46:20.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=191 2024-09-18T14:46:20.571 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:46:20.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:46:24.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:46:24.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:46:24.576 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:46:24.577 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:46:24.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:24.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:46:24.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:46:24.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:46:24.581 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:24.582 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:24.582 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:24.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:46:24.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:24.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:46:24.674 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:24.674 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:24.674 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:24.674 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:24.674 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:24.684 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:24.684 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:46:24.685 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:46:24.685 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:46:24.685 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:24.685 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:24.685 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:24.686 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:46:24.686 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:24.687 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:46:24.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:46:24.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:46:24.782 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 3 2024-09-18T14:46:24.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=3 2024-09-18T14:46:24.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:24.785 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:46:24.785 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:46:24.785 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:46:24.785 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:24.786 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:24.786 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:24.786 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:46:24.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:24.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:46:24.873 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:24.873 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:46:24.873 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 4, 2024-09-18T14:46:24.873 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:24.873 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:24.882 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:24.883 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:46:24.883 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:46:24.883 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:46:24.883 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:24.883 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:24.883 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:24.884 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:46:24.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:24.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:46:24.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:24.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:24.979 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 3 + 4 2024-09-18T14:46:24.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=7 2024-09-18T14:46:24.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:24.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:46:24.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:46:24.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:46:24.982 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:24.982 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:24.983 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:24.983 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:46:24.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:24.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:46:25.071 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:25.071 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:46:25.072 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:46:25.072 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:25.072 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:25.080 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:25.081 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:46:25.081 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:46:25.081 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:46:25.081 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:25.081 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:25.082 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:25.082 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:46:25.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:25.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:46:25.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:46:25.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:46:25.180 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 7 + 2 2024-09-18T14:46:25.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=9 2024-09-18T14:46:25.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:25.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:46:25.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:46:25.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:46:25.183 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:25.183 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:25.183 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:25.183 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:46:25.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:25.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:46:25.272 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:25.272 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:46:25.272 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:25.272 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:25.272 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:25.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:25.281 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:46:25.282 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:46:25.282 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:46:25.282 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:25.282 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:25.282 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:25.283 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:46:25.283 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:25.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:46:25.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:25.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:25.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 9 + 4 2024-09-18T14:46:25.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=13 2024-09-18T14:46:25.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:25.375 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:46:25.375 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:46:25.375 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:46:25.376 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:25.376 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:25.376 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:25.376 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:46:25.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:25.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:46:25.463 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:25.463 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:25.463 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:46:25.463 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:25.463 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:25.472 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:25.473 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:46:25.473 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:46:25.473 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:46:25.473 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:25.473 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:25.473 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:25.474 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:46:25.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:25.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:46:25.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:46:25.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:46:25.572 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 13 + 3 2024-09-18T14:46:25.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=16 2024-09-18T14:46:25.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:25.574 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:46:25.574 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:46:25.574 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:46:25.575 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:25.575 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:25.575 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:25.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:46:25.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:25.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:46:25.656 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:25.657 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:25.657 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:46:25.657 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:25.657 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:25.666 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:25.666 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:46:25.666 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:46:25.666 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:46:25.667 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:25.667 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:25.667 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:25.668 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:46:25.668 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:25.668 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:46:25.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:25.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:25.762 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 16 + 4 2024-09-18T14:46:25.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=20 2024-09-18T14:46:25.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 20 = 0 ']' 2024-09-18T14:46:25.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:46:25.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 191 + 20 2024-09-18T14:46:25.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=211 2024-09-18T14:46:25.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:46:25.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:46:29.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:46:29.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:46:29.770 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:46:29.772 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:46:29.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:29.775 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:46:29.775 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:46:29.775 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:46:29.776 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:29.776 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:29.777 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:29.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:46:29.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:29.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:46:29.868 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:29.869 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:46:29.869 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:29.869 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:29.869 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:29.878 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:29.879 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:46:29.879 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:46:29.879 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:46:29.879 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:29.880 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:29.880 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:29.880 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:46:29.880 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:29.880 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:46:29.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:46:29.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:46:29.976 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 2 2024-09-18T14:46:29.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=2 2024-09-18T14:46:29.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:29.978 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:46:29.978 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:46:29.978 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:46:29.979 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:29.979 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:29.980 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:29.980 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:46:29.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:29.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:46:30.061 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:30.061 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:30.061 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:46:30.061 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:30.061 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:30.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:30.070 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:46:30.071 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:46:30.071 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:46:30.071 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:30.071 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:30.071 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:30.072 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:46:30.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:30.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:46:30.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:30.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:30.168 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 2 + 4 2024-09-18T14:46:30.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=6 2024-09-18T14:46:30.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:30.170 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:46:30.170 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:46:30.170 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:46:30.171 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:30.171 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:30.171 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:30.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:46:30.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:30.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:46:30.261 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:30.261 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:30.261 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:30.261 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:30.261 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:30.271 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:30.271 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:46:30.271 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:46:30.272 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:46:30.272 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:30.272 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:30.272 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:30.272 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:46:30.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:30.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:46:30.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:46:30.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:46:30.367 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 6 + 3 2024-09-18T14:46:30.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=9 2024-09-18T14:46:30.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:30.370 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:46:30.370 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:46:30.370 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:46:30.371 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:30.371 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:30.371 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:30.372 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:46:30.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:30.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:46:30.456 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:30.456 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:46:30.457 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:30.457 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:30.457 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:30.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:30.466 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:46:30.466 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:46:30.466 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:46:30.467 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:30.467 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:30.467 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:30.467 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:46:30.468 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:30.468 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:46:30.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:30.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:30.565 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 9 + 4 2024-09-18T14:46:30.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=13 2024-09-18T14:46:30.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:30.568 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:46:30.568 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:46:30.568 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:46:30.569 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:30.569 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:30.569 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:30.569 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:46:30.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:30.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:46:30.658 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:30.658 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:46:30.658 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:30.659 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:30.659 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:30.668 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:30.668 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:46:30.668 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:46:30.669 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:46:30.671 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:30.672 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:30.672 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:30.673 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:46:30.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:30.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:46:30.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:30.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:30.766 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 13 + 4 2024-09-18T14:46:30.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=17 2024-09-18T14:46:30.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:30.769 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:46:30.769 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:46:30.769 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:46:30.769 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:30.770 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:30.770 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:30.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:46:30.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:30.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:46:30.856 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:30.856 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:30.856 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:46:30.856 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:30.856 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:30.866 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:30.866 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:46:30.866 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:46:30.866 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:46:30.867 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:30.867 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:30.867 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:30.867 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:46:30.868 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:30.868 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:46:30.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:30.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:30.962 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 17 + 4 2024-09-18T14:46:30.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=21 2024-09-18T14:46:30.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 21 = 0 ']' 2024-09-18T14:46:30.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:46:30.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 211 + 21 2024-09-18T14:46:30.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=232 2024-09-18T14:46:30.966 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:46:30.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:46:34.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:46:34.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:46:34.972 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:46:34.973 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:46:34.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:34.976 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:46:34.976 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:46:34.977 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:46:34.977 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:34.977 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:34.977 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:34.978 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:46:34.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:34.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:46:35.072 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:35.072 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:46:35.072 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:35.072 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:35.072 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:35.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:35.083 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:46:35.083 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:46:35.083 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:46:35.085 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:35.085 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:35.085 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:35.085 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:46:35.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:35.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:46:35.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:46:35.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:46:35.180 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 2 2024-09-18T14:46:35.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=2 2024-09-18T14:46:35.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:35.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:46:35.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:46:35.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:46:35.184 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:35.184 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:35.184 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:35.184 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:46:35.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:35.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:46:35.270 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:35.270 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:35.270 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:46:35.270 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:35.270 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:35.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:35.280 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:46:35.280 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:46:35.280 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:46:35.280 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:35.280 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:35.280 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:35.280 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:46:35.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:35.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:46:35.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:35.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:35.377 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 2 + 4 2024-09-18T14:46:35.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=6 2024-09-18T14:46:35.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:35.381 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:46:35.381 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:46:35.381 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:46:35.381 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:35.381 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:35.381 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:35.381 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:46:35.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:35.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:46:35.465 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:35.465 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:35.465 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:35.466 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:35.466 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:35.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:35.475 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:46:35.475 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:46:35.475 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:46:35.476 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:35.476 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:35.476 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:35.476 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:46:35.477 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:35.477 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:46:35.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:46:35.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:46:35.572 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 6 + 3 2024-09-18T14:46:35.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=9 2024-09-18T14:46:35.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:35.574 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:46:35.574 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:46:35.574 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:46:35.574 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:35.575 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:35.575 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:35.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:46:35.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:35.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:46:35.660 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:35.661 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:46:35.661 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:35.661 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:35.661 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:35.670 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:35.670 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:46:35.670 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:46:35.671 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:46:35.671 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:35.671 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:35.671 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:35.671 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:46:35.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:35.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:46:35.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:35.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:35.769 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 9 + 4 2024-09-18T14:46:35.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=13 2024-09-18T14:46:35.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:35.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:46:35.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:46:35.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:46:35.772 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:35.772 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:35.772 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:35.772 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:46:35.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:35.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:46:35.857 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:35.857 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:46:35.857 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:35.857 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:35.857 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:35.866 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:35.867 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:46:35.867 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:46:35.867 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:46:35.867 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:35.867 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:35.868 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:35.868 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:46:35.868 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:35.868 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:46:35.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:35.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:35.976 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 13 + 4 2024-09-18T14:46:35.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=17 2024-09-18T14:46:35.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:35.979 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:46:35.979 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:46:35.979 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:46:35.980 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:35.980 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:35.980 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:35.980 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:46:35.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:35.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:46:36.062 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:36.062 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:36.062 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:46:36.062 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:36.063 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:36.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:36.072 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:46:36.072 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:46:36.072 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:46:36.073 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:36.073 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:36.073 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:36.073 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:46:36.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:36.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:46:36.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:36.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:36.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 17 + 4 2024-09-18T14:46:36.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=21 2024-09-18T14:46:36.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 21 = 0 ']' 2024-09-18T14:46:36.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:46:36.162 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 232 + 21 2024-09-18T14:46:36.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=253 2024-09-18T14:46:36.165 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:46:36.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:46:40.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:46:40.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:46:40.170 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:46:40.172 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:46:40.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:40.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:46:40.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:46:40.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:46:40.176 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:40.176 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:40.176 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:40.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:46:40.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:40.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:46:40.269 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:40.269 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:46:40.269 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:46:40.269 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:40.269 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:40.278 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:40.279 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:46:40.279 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:46:40.279 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:46:40.280 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:40.280 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:40.280 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:40.280 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:46:40.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:40.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:46:40.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:46:40.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:46:40.372 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 1 2024-09-18T14:46:40.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=1 2024-09-18T14:46:40.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:40.374 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:46:40.374 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:46:40.374 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:46:40.375 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:40.375 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:40.375 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:40.375 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:46:40.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:40.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:46:40.463 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:40.463 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:40.463 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:46:40.463 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:40.463 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:40.473 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:40.473 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:46:40.473 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:46:40.473 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:46:40.473 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:40.473 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:40.473 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:40.473 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:46:40.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:40.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:46:40.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:40.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:40.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 1 + 4 2024-09-18T14:46:40.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=5 2024-09-18T14:46:40.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:40.569 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:46:40.569 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:46:40.569 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:46:40.570 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:40.570 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:40.570 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:40.570 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:46:40.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:40.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:46:40.657 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:40.657 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:40.658 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:40.658 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:40.658 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:40.667 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:40.667 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:46:40.667 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:46:40.667 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:46:40.668 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:40.668 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:40.668 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:40.668 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:46:40.669 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:40.669 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:46:40.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:46:40.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:46:40.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 5 + 3 2024-09-18T14:46:40.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=8 2024-09-18T14:46:40.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:40.768 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:46:40.768 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:46:40.768 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:46:40.769 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:40.769 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:40.769 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:40.769 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:46:40.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:40.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:46:40.855 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:40.855 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:40.856 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:40.856 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:40.856 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:40.864 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:40.865 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:46:40.865 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:46:40.865 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:46:40.866 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:40.866 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:40.866 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:40.867 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:46:40.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:40.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:46:40.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:46:40.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:46:40.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 8 + 3 2024-09-18T14:46:40.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=11 2024-09-18T14:46:40.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:40.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:46:40.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:46:40.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:46:40.964 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:40.964 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:40.964 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:40.965 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:46:40.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:40.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:46:41.051 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:41.051 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:46:41.051 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:41.051 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:41.052 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:41.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:41.061 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:46:41.061 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:46:41.061 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:46:41.062 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:41.062 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:41.062 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:41.063 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:46:41.063 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:41.063 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:46:41.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:41.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:41.158 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 11 + 4 2024-09-18T14:46:41.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=15 2024-09-18T14:46:41.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:41.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:46:41.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:46:41.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:46:41.162 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:41.162 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:41.162 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:41.162 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:46:41.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:41.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:46:41.244 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:41.244 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:41.244 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:41.244 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:41.244 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:41.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:41.254 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:46:41.254 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:46:41.254 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:46:41.254 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:41.254 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:41.254 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:41.255 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:46:41.256 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:41.256 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:46:41.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:46:41.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:46:41.353 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 15 + 3 2024-09-18T14:46:41.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=18 2024-09-18T14:46:41.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 18 = 0 ']' 2024-09-18T14:46:41.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:46:41.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 253 + 18 2024-09-18T14:46:41.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=271 2024-09-18T14:46:41.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:46:41.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:46:45.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:46:45.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:46:45.363 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:46:45.364 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:46:45.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:45.367 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:46:45.367 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:46:45.367 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:46:45.368 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:45.368 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:45.368 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:45.369 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:46:45.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:45.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:46:45.462 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:45.462 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:46:45.462 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:46:45.462 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:45.462 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:45.472 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:45.473 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:46:45.473 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:46:45.473 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:46:45.474 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:45.474 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:45.474 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:45.474 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:46:45.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:45.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:46:45.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:46:45.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:46:45.568 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 1 2024-09-18T14:46:45.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=1 2024-09-18T14:46:45.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:45.570 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:46:45.570 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:46:45.570 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:46:45.571 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:45.571 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:45.571 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:45.571 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:46:45.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:45.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:46:45.657 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:45.657 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:45.657 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:46:45.657 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:45.657 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:45.667 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:45.667 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:46:45.667 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:46:45.667 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:46:45.667 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:45.667 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:45.667 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:45.668 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:46:45.668 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:45.668 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:46:45.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:45.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:45.762 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 1 + 4 2024-09-18T14:46:45.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=5 2024-09-18T14:46:45.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:45.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:46:45.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:46:45.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:46:45.765 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:45.765 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:45.765 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:45.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:46:45.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:45.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:46:45.853 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:45.854 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:45.854 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:45.854 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:45.854 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:45.864 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:45.864 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:46:45.864 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:46:45.864 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:46:45.864 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:45.865 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:45.865 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:45.865 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:46:45.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:45.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:46:45.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:46:45.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:46:45.962 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 5 + 3 2024-09-18T14:46:45.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=8 2024-09-18T14:46:45.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:45.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:46:45.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:46:45.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:46:45.965 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:45.965 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:45.965 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:45.965 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:46:45.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:45.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:46:46.053 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:46.053 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:46.054 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:46.054 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:46.054 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:46.063 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:46.063 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:46:46.063 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:46:46.064 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:46:46.064 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:46.064 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:46.064 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:46.064 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:46:46.065 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:46.065 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:46:46.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:46:46.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:46:46.159 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 8 + 3 2024-09-18T14:46:46.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=11 2024-09-18T14:46:46.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:46.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:46:46.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:46:46.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:46:46.162 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:46.162 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:46.162 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:46.162 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:46:46.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:46.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:46:46.249 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:46.249 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:46:46.249 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:46.249 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:46.249 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:46.258 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:46.259 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:46:46.259 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:46:46.259 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:46:46.260 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:46.260 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:46.260 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:46.260 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:46:46.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:46.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:46:46.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:46.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:46.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 11 + 4 2024-09-18T14:46:46.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=15 2024-09-18T14:46:46.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:46.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:46:46.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:46:46.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:46:46.361 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:46.361 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:46.362 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:46.362 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:46:46.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:46.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:46:46.448 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:46.449 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:46.449 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:46.449 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:46.449 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:46.458 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:46.458 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:46:46.458 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:46:46.459 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:46:46.459 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:46.459 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:46.459 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:46.460 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:46:46.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:46.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:46:46.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:46:46.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:46:46.555 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 15 + 3 2024-09-18T14:46:46.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=18 2024-09-18T14:46:46.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 18 = 0 ']' 2024-09-18T14:46:46.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:46:46.558 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 271 + 18 2024-09-18T14:46:46.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=289 2024-09-18T14:46:46.560 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:46:46.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:46:50.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:46:50.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:46:50.566 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:46:50.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:46:50.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:50.570 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:46:50.570 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:46:50.570 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:46:50.571 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:50.571 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:50.571 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:50.572 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:46:50.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:50.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:46:50.668 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:50.668 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:46:50.668 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:46:50.668 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:50.669 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:50.678 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:50.679 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:46:50.679 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:46:50.679 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:46:50.680 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:50.680 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:50.680 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:50.680 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:46:50.681 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:50.681 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:46:50.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:46:50.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:46:50.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 3 2024-09-18T14:46:50.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=3 2024-09-18T14:46:50.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:50.779 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:46:50.779 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:46:50.779 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:46:50.779 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:50.780 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:50.780 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:50.780 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:46:50.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:50.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:46:50.865 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:50.865 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:50.865 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:50.865 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:50.865 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:50.874 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:50.875 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:46:50.875 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:46:50.875 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:46:50.876 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:50.876 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:50.876 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:50.876 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:46:50.877 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:50.877 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:46:50.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:46:50.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:46:50.970 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 3 + 3 2024-09-18T14:46:50.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=6 2024-09-18T14:46:50.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:50.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:46:50.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:46:50.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:46:50.973 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:50.973 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:50.973 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:50.974 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:46:50.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:50.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:46:51.060 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:51.060 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:46:51.060 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:51.060 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:51.061 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:51.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:51.070 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:46:51.070 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:46:51.070 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:46:51.071 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:51.071 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:51.071 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:51.072 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:46:51.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:51.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:46:51.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:46:51.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:46:51.166 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 6 + 2 2024-09-18T14:46:51.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=8 2024-09-18T14:46:51.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:51.168 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:46:51.169 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:46:51.169 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:46:51.169 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:51.169 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:51.169 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:51.170 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:46:51.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:51.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:46:51.256 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:51.256 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:46:51.256 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:51.256 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:51.256 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:51.265 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:51.265 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:46:51.266 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:46:51.266 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:46:51.266 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:51.267 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:51.267 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:51.267 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:46:51.268 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:51.268 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:46:51.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:46:51.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:46:51.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 8 + 2 2024-09-18T14:46:51.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=10 2024-09-18T14:46:51.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:51.366 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:46:51.366 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:46:51.366 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:46:51.366 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:51.367 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:51.367 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:51.367 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:46:51.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:51.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:46:51.453 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:51.453 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:51.453 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:46:51.453 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:51.453 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:51.463 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:51.464 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:46:51.464 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:46:51.464 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:46:51.464 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:51.465 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:51.465 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:51.465 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:46:51.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:51.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:46:51.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:51.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:51.559 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 10 + 4 2024-09-18T14:46:51.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=14 2024-09-18T14:46:51.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:51.561 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:46:51.561 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:46:51.561 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:46:51.561 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:51.561 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:51.562 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:51.562 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:46:51.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:51.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:46:51.648 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:51.649 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:46:51.649 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:51.649 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:51.649 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:51.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:51.658 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:46:51.658 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:46:51.658 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:46:51.659 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:51.659 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:51.659 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:51.659 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:46:51.660 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:51.660 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:46:51.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:51.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:51.755 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 14 + 4 2024-09-18T14:46:51.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=18 2024-09-18T14:46:51.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 18 = 0 ']' 2024-09-18T14:46:51.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:46:51.758 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 289 + 18 2024-09-18T14:46:51.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=307 2024-09-18T14:46:51.760 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:46:51.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:46:55.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:46:55.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:46:55.765 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:46:55.767 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:46:55.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:55.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:46:55.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:46:55.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:46:55.771 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:55.771 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:55.771 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:55.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:46:55.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:55.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:46:55.867 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:55.867 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:46:55.867 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:55.868 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:55.868 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:55.877 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:55.878 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:46:55.878 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:46:55.878 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:46:55.879 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:55.879 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:55.879 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:55.879 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:46:55.880 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:55.880 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:46:55.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:55.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:55.976 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 4 2024-09-18T14:46:55.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=4 2024-09-18T14:46:55.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:55.977 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:46:55.977 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:46:55.978 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:46:55.978 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:55.978 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:55.978 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:55.979 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:46:55.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:55.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:46:56.062 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:56.062 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:56.063 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:46:56.063 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:56.063 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:56.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:56.072 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:46:56.072 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:46:56.072 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:46:56.072 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:56.073 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:56.073 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:56.073 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:46:56.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:56.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:46:56.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:46:56.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:46:56.162 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 4 + 4 2024-09-18T14:46:56.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=8 2024-09-18T14:46:56.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:56.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:46:56.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:46:56.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:46:56.165 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:56.165 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:56.165 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:56.165 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:46:56.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:56.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:46:56.247 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:56.247 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:46:56.247 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:56.247 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:56.247 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:56.256 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:56.256 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:46:56.257 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:46:56.257 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:46:56.257 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:56.257 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:56.257 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:56.258 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:46:56.258 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:56.258 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:46:56.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:46:56.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:46:56.353 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 8 + 2 2024-09-18T14:46:56.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=10 2024-09-18T14:46:56.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:56.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:46:56.356 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:46:56.356 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:46:56.356 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:56.356 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:56.357 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:56.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:46:56.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:56.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:46:56.438 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:56.438 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:56.438 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:46:56.438 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:56.438 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:56.447 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:56.447 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:46:56.447 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:46:56.447 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:46:56.448 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:56.448 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:56.448 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:56.448 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:46:56.448 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:56.448 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:46:56.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:46:56.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:46:56.542 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 10 + 2 2024-09-18T14:46:56.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=12 2024-09-18T14:46:56.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:56.545 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:46:56.545 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:46:56.545 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:46:56.546 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:56.546 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:56.546 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:56.546 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:46:56.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:56.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:46:56.631 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:56.631 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:56.631 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:56.632 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:56.632 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:56.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:56.641 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:46:56.641 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:46:56.641 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:46:56.642 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:56.642 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:56.642 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:56.642 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:46:56.642 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:56.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:46:56.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:46:56.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:46:56.744 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 12 + 3 2024-09-18T14:46:56.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=15 2024-09-18T14:46:56.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:46:56.746 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:46:56.746 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:46:56.746 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:46:56.746 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:56.747 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:56.747 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:56.747 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:46:56.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:56.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:46:56.830 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:46:56.830 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:46:56.830 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:46:56.830 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:46:56.830 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:46:56.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:46:56.841 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:46:56.841 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:46:56.841 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:46:56.841 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:46:56.841 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:46:56.842 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:46:56.842 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:46:56.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:46:56.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:46:56.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:46:56.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:46:56.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 15 + 3 2024-09-18T14:46:56.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=18 2024-09-18T14:46:56.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 18 = 0 ']' 2024-09-18T14:46:56.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:46:56.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 307 + 18 2024-09-18T14:46:56.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=325 2024-09-18T14:46:56.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:46:56.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:47:00.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:47:00.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:47:00.941 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:47:00.942 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:47:00.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:00.945 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:47:00.945 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:47:00.945 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:47:00.946 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:00.946 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:00.946 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:00.946 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:47:00.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:00.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:47:01.041 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:01.041 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:47:01.041 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:01.041 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:01.041 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:01.051 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:01.052 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:47:01.052 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:47:01.052 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:47:01.052 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:01.052 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:01.052 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:01.052 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:47:01.052 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:01.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:47:01.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:01.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:01.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 4 2024-09-18T14:47:01.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=4 2024-09-18T14:47:01.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:01.150 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:47:01.150 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:47:01.151 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:47:01.151 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:01.151 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:01.151 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:01.152 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:47:01.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:01.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:47:01.237 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:01.238 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:47:01.238 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:47:01.238 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:01.238 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:01.247 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:01.247 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:47:01.248 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:47:01.248 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:47:01.248 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:01.248 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:01.248 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:01.249 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:47:01.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:01.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:47:01.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:47:01.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:47:01.343 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 4 + 3 2024-09-18T14:47:01.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=7 2024-09-18T14:47:01.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:01.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:47:01.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:47:01.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:47:01.347 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:01.347 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:01.347 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:01.347 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:47:01.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:01.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:47:01.433 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:01.434 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:47:01.434 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:01.434 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:01.434 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:01.443 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:01.444 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:47:01.444 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:47:01.444 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:47:01.444 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:01.444 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:01.444 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:01.444 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:47:01.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:01.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:47:01.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:47:01.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:47:01.538 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 7 + 2 2024-09-18T14:47:01.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=9 2024-09-18T14:47:01.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:01.540 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:47:01.540 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:47:01.540 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:47:01.541 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:01.541 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:01.541 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:01.541 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:47:01.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:01.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:47:01.627 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:01.627 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:47:01.627 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:01.627 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:01.628 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:01.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:01.637 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:47:01.637 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:47:01.637 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:47:01.638 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:01.638 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:01.638 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:01.638 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:47:01.639 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:01.639 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:47:01.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:01.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:01.732 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 9 + 4 2024-09-18T14:47:01.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=13 2024-09-18T14:47:01.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:01.734 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:47:01.734 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:47:01.734 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:47:01.735 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:01.735 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:01.735 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:01.736 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:47:01.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:01.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:47:01.821 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:01.821 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:47:01.821 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:01.822 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:01.822 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:01.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:01.831 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:47:01.831 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:47:01.831 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:47:01.832 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:01.832 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:01.832 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:01.832 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:47:01.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:01.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:47:01.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:01.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:01.927 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 13 + 4 2024-09-18T14:47:01.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=17 2024-09-18T14:47:01.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:01.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:47:01.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:47:01.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:47:01.930 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:01.931 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:01.931 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:01.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:47:01.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:01.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:47:02.018 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:02.018 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:02.018 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:47:02.018 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:02.019 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:02.027 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:02.027 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:47:02.028 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:47:02.028 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:47:02.029 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:02.029 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:02.029 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:02.029 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:47:02.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:02.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:47:02.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:02.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:02.125 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 17 + 4 2024-09-18T14:47:02.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=21 2024-09-18T14:47:02.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 21 = 0 ']' 2024-09-18T14:47:02.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:47:02.127 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 325 + 21 2024-09-18T14:47:02.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=346 2024-09-18T14:47:02.129 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:47:02.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:47:06.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:47:06.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:47:06.135 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:47:06.137 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:47:06.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:06.140 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:47:06.140 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:47:06.140 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:47:06.140 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:06.141 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:06.141 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:06.141 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:47:06.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:06.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:47:06.234 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:06.235 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:47:06.235 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:06.235 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:06.235 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:06.244 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:06.245 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:47:06.245 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:47:06.245 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:47:06.246 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:06.246 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:06.246 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:06.246 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:47:06.247 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:06.247 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:47:06.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:06.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:06.342 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 4 2024-09-18T14:47:06.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=4 2024-09-18T14:47:06.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:06.344 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:47:06.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:47:06.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:47:06.345 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:06.346 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:06.346 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:06.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:47:06.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:06.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:47:06.428 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:06.429 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:47:06.429 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:47:06.429 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:06.429 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:06.438 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:06.439 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:47:06.439 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:47:06.439 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:47:06.439 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:06.439 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:06.439 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:06.440 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:47:06.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:06.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:47:06.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:47:06.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:47:06.534 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 4 + 3 2024-09-18T14:47:06.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=7 2024-09-18T14:47:06.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:06.537 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:47:06.537 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:47:06.537 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:47:06.537 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:06.538 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:06.538 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:06.538 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:47:06.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:06.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:47:06.623 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:06.624 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:47:06.624 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:06.624 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:06.624 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:06.633 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:06.634 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:47:06.634 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:47:06.634 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:47:06.634 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:06.635 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:06.635 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:06.635 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:47:06.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:06.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:47:06.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:47:06.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:47:06.731 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 7 + 2 2024-09-18T14:47:06.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=9 2024-09-18T14:47:06.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:06.733 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:47:06.733 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:47:06.733 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:47:06.734 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:06.734 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:06.734 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:06.735 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:47:06.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:06.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:47:06.816 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:06.816 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:47:06.816 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:06.816 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:06.816 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:06.825 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:06.826 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:47:06.826 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:47:06.826 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:47:06.826 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:06.827 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:06.827 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:06.827 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:47:06.827 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:06.828 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:47:06.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:06.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:06.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 9 + 4 2024-09-18T14:47:06.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=13 2024-09-18T14:47:06.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:06.926 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:47:06.926 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:47:06.926 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:47:06.926 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:06.927 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:06.927 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:06.927 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:47:06.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:06.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:47:07.013 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:07.013 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:47:07.013 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:07.013 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:07.013 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:07.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:07.023 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:47:07.023 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:47:07.023 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:47:07.024 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:07.024 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:07.024 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:07.024 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:47:07.024 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:07.024 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:47:07.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:07.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:07.119 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 13 + 4 2024-09-18T14:47:07.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=17 2024-09-18T14:47:07.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:07.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:47:07.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:47:07.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:47:07.123 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:07.123 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:07.123 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:07.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:47:07.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:07.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:47:07.208 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:07.208 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:07.208 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:47:07.208 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:07.208 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:07.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:07.218 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:47:07.218 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:47:07.218 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:47:07.219 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:07.219 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:07.219 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:07.219 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:47:07.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:07.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:47:07.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:07.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:07.310 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 17 + 4 2024-09-18T14:47:07.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=21 2024-09-18T14:47:07.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 21 = 0 ']' 2024-09-18T14:47:07.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:47:07.313 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 346 + 21 2024-09-18T14:47:07.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=367 2024-09-18T14:47:07.315 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:47:07.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:47:11.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:47:11.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:47:11.321 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:47:11.323 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:47:11.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:11.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:47:11.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:47:11.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:47:11.327 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:11.327 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:11.327 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:11.327 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:47:11.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:11.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:47:11.422 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:11.422 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:47:11.422 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:47:11.422 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:11.422 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:11.432 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:11.432 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:47:11.432 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:47:11.432 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:47:11.433 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:11.433 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:11.433 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:11.434 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:47:11.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:11.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:47:11.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:47:11.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:47:11.525 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 3 2024-09-18T14:47:11.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=3 2024-09-18T14:47:11.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:11.529 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:47:11.529 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:47:11.529 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:47:11.529 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:11.529 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:11.529 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:11.529 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:47:11.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:11.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:47:11.616 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:11.616 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:47:11.616 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 4, 2024-09-18T14:47:11.617 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:11.617 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:11.625 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:11.626 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:47:11.626 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:47:11.626 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:47:11.627 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:11.627 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:11.627 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:11.628 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:47:11.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:11.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:47:11.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:11.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:11.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 3 + 4 2024-09-18T14:47:11.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=7 2024-09-18T14:47:11.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:11.723 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:47:11.724 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:47:11.724 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:47:11.724 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:11.724 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:11.725 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:11.725 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:47:11.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:11.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:47:11.811 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:11.811 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:47:11.811 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:11.812 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:11.812 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:11.820 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:11.821 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:47:11.821 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:47:11.821 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:47:11.821 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:11.822 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:11.822 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:11.822 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:47:11.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:11.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:47:11.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:47:11.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:47:11.918 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 7 + 2 2024-09-18T14:47:11.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=9 2024-09-18T14:47:11.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:11.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:47:11.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:47:11.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:47:11.921 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:11.921 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:11.921 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:11.921 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:47:11.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:11.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:47:12.007 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:12.007 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:47:12.007 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:12.007 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:12.007 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:12.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:12.017 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:47:12.017 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:47:12.017 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:47:12.021 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:12.021 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:12.021 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:12.022 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:47:12.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:12.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:47:12.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:12.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:12.116 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 9 + 4 2024-09-18T14:47:12.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=13 2024-09-18T14:47:12.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:12.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:47:12.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:47:12.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:47:12.119 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:12.119 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:12.119 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:12.119 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:47:12.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:12.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:47:12.203 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:12.203 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:12.204 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:47:12.204 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:12.204 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:12.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:12.214 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:47:12.214 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:47:12.214 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:47:12.215 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:12.215 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:12.215 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:12.215 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:47:12.215 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:12.215 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:47:12.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:12.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:12.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 13 + 4 2024-09-18T14:47:12.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=17 2024-09-18T14:47:12.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:12.306 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:47:12.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:47:12.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:47:12.307 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:12.307 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:12.307 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:12.308 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:47:12.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:12.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:47:12.389 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:12.389 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:12.389 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:12.389 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:12.389 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:12.399 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:12.400 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:47:12.400 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:47:12.400 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:47:12.400 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:12.400 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:12.400 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:12.400 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:47:12.401 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:12.401 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:47:12.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:47:12.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:47:12.494 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 17 + 3 2024-09-18T14:47:12.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=20 2024-09-18T14:47:12.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 20 = 0 ']' 2024-09-18T14:47:12.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:47:12.496 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 367 + 20 2024-09-18T14:47:12.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=387 2024-09-18T14:47:12.498 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:47:12.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:47:16.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:47:16.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:47:16.504 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:47:16.505 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:47:16.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:16.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:47:16.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:47:16.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:47:16.509 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:16.509 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:16.509 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:16.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:47:16.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:16.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:47:16.604 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:16.604 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:47:16.604 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:47:16.604 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:16.604 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:16.614 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:16.614 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:47:16.614 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:47:16.615 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:47:16.615 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:16.615 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:16.615 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:16.616 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:47:16.616 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:16.616 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:47:16.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:47:16.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:47:16.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 2 2024-09-18T14:47:16.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=2 2024-09-18T14:47:16.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:16.715 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:47:16.715 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:47:16.716 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:47:16.716 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:16.716 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:16.716 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:16.716 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:47:16.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:16.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:47:16.798 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:16.798 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:47:16.798 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:16.798 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:16.798 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:16.808 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:16.808 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:47:16.808 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:47:16.808 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:47:16.809 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:16.809 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:16.809 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:16.809 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:47:16.809 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:16.809 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:47:16.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:47:16.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:47:16.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 2 + 3 2024-09-18T14:47:16.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=5 2024-09-18T14:47:16.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:16.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:47:16.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:47:16.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:47:16.911 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:16.912 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:16.912 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:16.912 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:47:16.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:16.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:47:17.005 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:17.006 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:47:17.006 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:47:17.006 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:17.006 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:17.016 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:17.016 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:47:17.016 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:47:17.016 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:47:17.016 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:17.016 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:17.016 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:17.016 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:47:17.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:17.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:47:17.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:47:17.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:47:17.112 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 5 + 1 2024-09-18T14:47:17.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=6 2024-09-18T14:47:17.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:17.114 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:47:17.115 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:47:17.115 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:47:17.115 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:17.115 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:17.115 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:17.116 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:47:17.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:17.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:47:17.199 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:17.199 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:47:17.199 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:47:17.199 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:17.199 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:17.209 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:17.210 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:47:17.210 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:47:17.210 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:47:17.211 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:17.211 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:17.211 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:17.211 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:47:17.212 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:17.212 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:47:17.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:47:17.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:47:17.306 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 6 + 3 2024-09-18T14:47:17.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=9 2024-09-18T14:47:17.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:17.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:47:17.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:47:17.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:47:17.310 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:17.310 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:17.310 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:17.310 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:47:17.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:17.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:47:17.399 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:17.399 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:17.399 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:17.400 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:17.400 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:17.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:17.409 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:47:17.409 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:47:17.409 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:47:17.410 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:17.410 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:17.410 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:17.410 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:47:17.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:17.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:47:17.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:17.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:17.505 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 9 + 4 2024-09-18T14:47:17.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=13 2024-09-18T14:47:17.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:17.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:47:17.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:47:17.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:47:17.509 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:17.509 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:17.509 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:17.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:47:17.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:17.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:47:17.597 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:17.597 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:17.598 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:47:17.598 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:17.598 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:17.607 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:17.608 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:47:17.608 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:47:17.608 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:47:17.609 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:17.610 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:17.610 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:17.610 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:47:17.611 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:17.611 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:47:17.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:47:17.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:47:17.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 13 + 2 2024-09-18T14:47:17.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=15 2024-09-18T14:47:17.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 15 = 0 ']' 2024-09-18T14:47:17.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:47:17.707 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 387 + 15 2024-09-18T14:47:17.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=402 2024-09-18T14:47:17.710 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:47:17.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:47:21.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:47:21.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:47:21.716 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:47:21.718 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:47:21.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:21.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:47:21.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:47:21.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:47:21.722 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:21.722 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:21.722 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:21.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:47:21.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:21.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:47:21.818 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:21.818 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:47:21.818 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:21.819 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:21.819 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:21.828 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:21.828 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:47:21.828 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:47:21.829 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:47:21.829 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:21.829 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:21.829 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:21.830 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:47:21.830 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:21.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:47:21.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:21.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:21.925 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 4 2024-09-18T14:47:21.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=4 2024-09-18T14:47:21.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:21.927 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:47:21.927 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:47:21.927 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:47:21.928 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:21.928 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:21.928 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:21.928 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:47:21.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:21.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:47:22.016 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:22.016 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:47:22.016 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 4, 2024-09-18T14:47:22.016 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:22.016 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:22.025 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:22.026 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:47:22.026 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:47:22.026 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:47:22.027 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:22.027 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:22.027 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:22.027 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:47:22.028 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:22.028 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:47:22.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:22.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:22.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 4 + 4 2024-09-18T14:47:22.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=8 2024-09-18T14:47:22.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:22.124 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:47:22.124 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:47:22.125 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:47:22.125 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:22.125 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:22.125 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:22.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:47:22.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:22.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:47:22.206 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:22.206 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:47:22.206 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:47:22.206 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:22.206 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:22.215 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:22.215 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:47:22.215 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:47:22.215 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:47:22.216 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:22.216 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:22.216 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:22.216 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:47:22.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:22.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:47:22.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:47:22.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:47:22.310 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 8 + 1 2024-09-18T14:47:22.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=9 2024-09-18T14:47:22.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:22.313 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:47:22.313 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:47:22.313 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:47:22.313 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:22.313 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:22.314 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:22.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:47:22.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:22.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:47:22.404 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:22.404 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:47:22.404 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:47:22.405 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:22.405 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:22.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:22.414 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:47:22.414 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:47:22.414 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:47:22.415 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:22.415 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:22.415 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:22.415 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:47:22.416 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:22.416 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:47:22.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:47:22.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:47:22.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 9 + 3 2024-09-18T14:47:22.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=12 2024-09-18T14:47:22.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:22.515 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:47:22.515 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:47:22.515 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:47:22.516 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:22.516 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:22.516 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:22.516 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:47:22.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:22.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:47:22.597 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:22.598 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:22.598 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:47:22.598 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:22.598 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:22.607 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:22.607 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:47:22.607 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:47:22.607 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:47:22.608 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:22.608 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:22.608 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:22.608 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:47:22.609 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:22.609 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:47:22.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:22.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:22.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 12 + 4 2024-09-18T14:47:22.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=16 2024-09-18T14:47:22.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:22.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:47:22.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:47:22.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:47:22.702 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:22.702 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:22.702 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:22.702 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:47:22.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:22.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:47:22.784 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:22.785 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:22.785 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:47:22.785 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:22.785 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:22.794 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:22.794 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:47:22.795 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:47:22.795 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:47:22.795 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:22.795 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:22.796 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:22.796 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:47:22.797 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:22.797 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:47:22.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:47:22.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:47:22.891 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 16 + 2 2024-09-18T14:47:22.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=18 2024-09-18T14:47:22.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 18 = 0 ']' 2024-09-18T14:47:22.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:47:22.893 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 402 + 18 2024-09-18T14:47:22.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=420 2024-09-18T14:47:22.895 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:47:22.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:47:26.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:47:26.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:47:26.901 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:47:26.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:47:26.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:26.906 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:47:26.906 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:47:26.906 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:47:26.906 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:26.907 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:26.907 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:26.907 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:47:26.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:26.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:47:27.000 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:27.000 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:47:27.000 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:47:27.000 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:27.001 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:27.011 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:27.011 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:47:27.012 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:47:27.012 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:47:27.012 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:27.013 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:27.013 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:27.013 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:47:27.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:27.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:47:27.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:47:27.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:47:27.109 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 3 2024-09-18T14:47:27.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=3 2024-09-18T14:47:27.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:27.112 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:47:27.112 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:47:27.112 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:47:27.112 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:27.112 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:27.112 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:27.113 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:47:27.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:27.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:47:27.200 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:27.200 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:27.200 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:47:27.201 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:27.201 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:27.210 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:27.210 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:47:27.210 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:47:27.210 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:47:27.211 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:27.211 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:27.211 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:27.211 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:47:27.212 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:27.212 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:47:27.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:27.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:27.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 3 + 4 2024-09-18T14:47:27.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=7 2024-09-18T14:47:27.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:27.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:47:27.310 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:47:27.310 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:47:27.310 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:27.310 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:27.310 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:27.310 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:47:27.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:27.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:47:27.395 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:27.396 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:47:27.396 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:47:27.396 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:27.396 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:27.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:27.405 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:47:27.406 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:47:27.406 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:47:27.407 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:27.407 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:27.407 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:27.407 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:47:27.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:27.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:47:27.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:47:27.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:47:27.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 7 + 1 2024-09-18T14:47:27.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=8 2024-09-18T14:47:27.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:27.503 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:47:27.503 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:47:27.503 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:47:27.504 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:27.504 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:27.504 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:27.504 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:47:27.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:27.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:47:27.588 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:27.588 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:47:27.588 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:47:27.588 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:27.589 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:27.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:27.598 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:47:27.598 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:47:27.598 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:47:27.599 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:27.599 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:27.599 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:27.599 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:47:27.600 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:27.600 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:47:27.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:47:27.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:47:27.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 8 + 0 2024-09-18T14:47:27.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=8 2024-09-18T14:47:27.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:27.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:47:27.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:47:27.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:47:27.694 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:27.694 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:27.694 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:27.694 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:47:27.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:27.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:47:27.777 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:27.778 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:27.778 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:47:27.778 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:27.778 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:27.788 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:27.788 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:47:27.788 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:47:27.788 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:47:27.788 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:27.788 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:27.788 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:27.788 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:47:27.789 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:27.789 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:47:27.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:27.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:27.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 8 + 4 2024-09-18T14:47:27.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=12 2024-09-18T14:47:27.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:27.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:47:27.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:47:27.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:47:27.889 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:27.889 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:27.889 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:27.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:47:27.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:27.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:47:27.974 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:27.974 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:27.974 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:27.975 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:27.975 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:27.984 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:27.984 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:47:27.984 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:47:27.984 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:47:27.984 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:27.984 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:27.984 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:27.984 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:47:27.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:27.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:47:28.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:47:28.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:47:28.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 12 + 3 2024-09-18T14:47:28.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=15 2024-09-18T14:47:28.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 15 = 0 ']' 2024-09-18T14:47:28.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:47:28.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 420 + 15 2024-09-18T14:47:28.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=435 2024-09-18T14:47:28.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:47:28.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:47:32.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:47:32.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:47:32.091 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:47:32.093 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:47:32.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:32.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:47:32.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:47:32.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:47:32.097 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:32.097 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:32.097 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:32.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:47:32.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:32.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:47:32.192 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:32.192 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:47:32.192 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:47:32.192 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:32.192 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:32.202 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:32.202 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:47:32.202 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:47:32.202 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:47:32.203 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:32.203 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:32.203 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:32.204 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:47:32.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:32.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:47:32.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:47:32.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:47:32.302 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 3 2024-09-18T14:47:32.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=3 2024-09-18T14:47:32.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:32.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:47:32.305 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:47:32.305 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:47:32.305 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:32.305 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:32.306 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:32.306 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:47:32.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:32.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:47:32.393 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:32.393 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:32.393 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:47:32.393 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:32.393 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:32.402 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:32.402 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:47:32.402 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:47:32.403 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:47:32.403 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:32.403 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:32.403 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:32.403 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:47:32.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:32.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:47:32.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:32.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:32.499 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 3 + 4 2024-09-18T14:47:32.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=7 2024-09-18T14:47:32.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:32.502 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:47:32.503 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:47:32.503 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:47:32.503 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:32.503 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:32.503 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:32.504 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:47:32.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:32.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:47:32.587 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:32.588 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:47:32.588 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:47:32.588 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:32.588 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:32.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:32.597 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:47:32.597 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:47:32.598 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:47:32.598 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:32.598 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:32.598 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:32.599 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:47:32.600 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:32.600 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:47:32.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:47:32.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:47:32.695 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 7 + 1 2024-09-18T14:47:32.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=8 2024-09-18T14:47:32.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:32.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:47:32.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:47:32.699 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:47:32.699 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:32.700 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:32.700 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:32.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:47:32.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:32.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:47:32.789 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:32.789 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:47:32.789 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:47:32.789 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:32.789 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:32.799 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:32.799 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:47:32.799 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:47:32.799 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:47:32.800 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:32.800 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:32.800 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:32.800 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:47:32.801 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:32.801 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:47:32.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:47:32.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:47:32.891 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 8 + 0 2024-09-18T14:47:32.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=8 2024-09-18T14:47:32.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:32.893 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:47:32.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:47:32.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:47:32.894 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:32.895 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:32.895 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:32.895 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:47:32.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:32.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:47:32.993 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:32.993 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:32.993 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:47:32.994 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:32.994 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:33.003 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:33.003 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:47:33.003 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:47:33.004 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:47:33.004 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:33.004 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:33.004 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:33.005 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:47:33.005 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:33.005 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:47:33.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:33.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:33.101 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 8 + 4 2024-09-18T14:47:33.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=12 2024-09-18T14:47:33.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:33.103 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:47:33.103 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:47:33.103 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:47:33.104 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:33.104 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:33.104 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:33.104 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:47:33.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:33.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:47:33.190 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:33.191 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:33.191 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:33.191 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:33.191 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:33.199 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:33.200 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:47:33.200 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:47:33.200 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:47:33.201 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:33.201 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:33.201 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:33.202 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:47:33.202 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:33.202 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:47:33.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:47:33.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:47:33.298 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 12 + 3 2024-09-18T14:47:33.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=15 2024-09-18T14:47:33.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 15 = 0 ']' 2024-09-18T14:47:33.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:47:33.301 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 435 + 15 2024-09-18T14:47:33.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=450 2024-09-18T14:47:33.303 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:47:33.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:47:37.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:47:37.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:47:37.309 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:47:37.310 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:47:37.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:37.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:47:37.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:47:37.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:47:37.314 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:37.314 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:37.315 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:37.315 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:47:37.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:37.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:47:37.408 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:37.408 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:37.408 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:47:37.408 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:37.408 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:37.418 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:37.418 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:47:37.418 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:47:37.418 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:47:37.419 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:37.419 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:37.419 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:37.420 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:47:37.420 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:37.420 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:47:37.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:47:37.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:47:37.516 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 2 2024-09-18T14:47:37.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=2 2024-09-18T14:47:37.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:37.517 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:47:37.518 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:47:37.518 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:47:37.518 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:37.518 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:37.519 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:37.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:47:37.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:37.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:47:37.599 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:37.600 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:47:37.600 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:37.600 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:37.600 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:37.609 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:37.609 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:47:37.609 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:47:37.609 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:47:37.610 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:37.610 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:37.610 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:37.611 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:47:37.611 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:37.611 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:47:37.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:37.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:37.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 2 + 4 2024-09-18T14:47:37.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=6 2024-09-18T14:47:37.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:37.704 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:47:37.704 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:47:37.704 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:47:37.705 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:37.705 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:37.705 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:37.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:47:37.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:37.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:47:37.788 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:37.789 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:47:37.789 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:37.789 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:37.789 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:37.798 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:37.798 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:47:37.798 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:47:37.799 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:47:37.799 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:37.799 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:37.799 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:37.800 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:47:37.800 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:37.800 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:47:37.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:47:37.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:47:37.896 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 6 + 2 2024-09-18T14:47:37.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=8 2024-09-18T14:47:37.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:37.899 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:47:37.899 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:47:37.899 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:47:37.899 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:37.900 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:37.900 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:37.900 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:47:37.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:37.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:47:37.988 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:37.988 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:47:37.988 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:47:37.988 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:37.988 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:37.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:37.998 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:47:37.998 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:47:37.998 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:47:37.999 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:37.999 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:37.999 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:37.999 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:47:38.000 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:38.000 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:47:38.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:47:38.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:47:38.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 8 + 0 2024-09-18T14:47:38.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=8 2024-09-18T14:47:38.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:38.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:47:38.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:47:38.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:47:38.097 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:38.097 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:38.097 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:38.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:47:38.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:38.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:47:38.183 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:38.183 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:38.183 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:47:38.183 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:38.183 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:38.192 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:38.192 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:47:38.192 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:47:38.192 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:47:38.193 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:38.193 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:38.193 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:38.194 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:47:38.194 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:38.194 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:47:38.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:38.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:38.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 8 + 4 2024-09-18T14:47:38.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=12 2024-09-18T14:47:38.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:38.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:47:38.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:47:38.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:47:38.285 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:38.285 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:38.285 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:38.285 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:47:38.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:38.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:47:38.367 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:38.367 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:38.367 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:38.368 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:38.368 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:38.376 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:38.377 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:47:38.377 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:47:38.377 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:47:38.378 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:38.378 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:38.378 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:38.378 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:47:38.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:38.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:47:38.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:47:38.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:47:38.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 12 + 3 2024-09-18T14:47:38.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=15 2024-09-18T14:47:38.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 15 = 0 ']' 2024-09-18T14:47:38.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:47:38.477 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 450 + 15 2024-09-18T14:47:38.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=465 2024-09-18T14:47:38.479 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:47:38.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:47:42.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:47:42.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:47:42.484 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:47:42.485 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:47:42.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:42.488 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:47:42.488 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:47:42.488 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:47:42.489 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:42.489 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:42.489 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:42.490 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:47:42.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:42.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:47:42.584 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:42.584 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:42.584 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:47:42.584 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:42.584 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:42.594 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:42.594 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:47:42.595 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:47:42.595 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:47:42.595 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:42.595 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:42.595 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:42.596 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:47:42.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:42.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:47:42.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:47:42.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:47:42.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 1 2024-09-18T14:47:42.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=1 2024-09-18T14:47:42.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:42.694 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:47:42.694 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:47:42.694 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:47:42.694 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:42.694 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:42.694 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:42.694 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:47:42.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:42.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:47:42.780 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:42.780 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:47:42.780 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:42.780 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:42.780 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:42.789 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:42.791 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:47:42.791 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:47:42.791 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:47:42.791 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:42.791 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:42.791 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:42.791 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:47:42.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:42.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:47:42.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:42.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:42.886 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 1 + 4 2024-09-18T14:47:42.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=5 2024-09-18T14:47:42.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:42.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:47:42.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:47:42.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:47:42.890 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:42.890 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:42.890 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:42.890 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:47:42.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:42.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:47:42.974 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:42.974 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:47:42.974 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:42.974 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:42.974 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:42.984 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:42.984 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:47:42.984 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:47:42.984 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:47:42.985 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:42.985 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:42.985 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:42.985 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:47:42.986 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:42.986 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:47:43.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:47:43.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:47:43.076 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 5 + 2 2024-09-18T14:47:43.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=7 2024-09-18T14:47:43.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:43.078 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:47:43.078 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:47:43.078 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:47:43.079 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:43.079 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:43.079 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:43.079 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:47:43.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:43.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:47:43.167 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:43.168 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:43.168 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:47:43.168 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:43.168 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:43.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:43.177 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:47:43.177 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:47:43.177 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:47:43.177 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:43.177 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:43.178 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:43.178 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:47:43.178 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:43.178 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:47:43.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:47:43.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:47:43.269 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 7 + 1 2024-09-18T14:47:43.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=8 2024-09-18T14:47:43.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:43.272 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:47:43.272 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:47:43.272 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:47:43.273 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:43.273 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:43.273 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:43.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:47:43.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:43.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:47:43.358 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:43.358 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:43.358 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:47:43.358 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:43.358 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:43.367 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:43.367 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:47:43.368 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:47:43.368 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:47:43.368 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:43.368 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:43.369 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:43.369 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:47:43.369 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:43.369 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:47:43.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:43.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:43.462 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 8 + 4 2024-09-18T14:47:43.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=12 2024-09-18T14:47:43.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:43.465 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:47:43.465 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:47:43.465 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:47:43.466 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:43.466 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:43.466 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:43.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:47:43.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:43.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:47:43.554 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:43.554 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:47:43.555 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:47:43.555 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:43.555 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:43.565 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:43.565 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:47:43.565 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:47:43.565 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:47:43.565 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:43.565 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:43.565 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:43.565 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:47:43.566 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:43.566 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:47:43.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:47:43.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:47:43.659 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 12 + 3 2024-09-18T14:47:43.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=15 2024-09-18T14:47:43.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 15 = 0 ']' 2024-09-18T14:47:43.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:47:43.661 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 465 + 15 2024-09-18T14:47:43.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=480 2024-09-18T14:47:43.663 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:47:43.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:47:47.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:47:47.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:47:47.668 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:47:47.669 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:47:47.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:47.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:47:47.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:47:47.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:47:47.673 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:47.673 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:47.673 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:47.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:47:47.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:47.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:47:47.766 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:47.766 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:47:47.766 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:47:47.766 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:47.766 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:47.776 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:47.776 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:47:47.776 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:47:47.776 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:47:47.777 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:47.777 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:47.777 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:47.778 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:47:47.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:47.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:47:47.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:47:47.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:47:47.868 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 0 2024-09-18T14:47:47.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=0 2024-09-18T14:47:47.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:47.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:47:47.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:47:47.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:47:47.871 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:47.871 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:47.871 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:47.872 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:47:47.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:47.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:47:47.958 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:47.958 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:47:47.959 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:47.959 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:47.959 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:47.968 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:47.968 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:47:47.968 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:47:47.969 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:47:47.969 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:47.969 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:47.970 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:47.970 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:47:47.970 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:47.970 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:47:48.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:48.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:48.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 4 2024-09-18T14:47:48.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=4 2024-09-18T14:47:48.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:48.064 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:47:48.064 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:47:48.064 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:47:48.065 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:48.065 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:48.065 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:48.065 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:47:48.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:48.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:47:48.152 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:48.152 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:47:48.152 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:48.152 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:48.152 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:48.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:48.162 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:47:48.162 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:47:48.162 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:47:48.163 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:48.163 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:48.163 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:48.163 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:47:48.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:48.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:47:48.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:47:48.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:47:48.258 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 4 + 2 2024-09-18T14:47:48.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=6 2024-09-18T14:47:48.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:48.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:47:48.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:47:48.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:47:48.261 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:48.262 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:48.262 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:48.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:47:48.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:48.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:47:48.348 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:48.348 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:48.348 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:47:48.348 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:48.348 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:48.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:48.358 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:47:48.358 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:47:48.358 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:47:48.359 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:48.359 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:48.359 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:48.359 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:47:48.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:48.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:47:48.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:47:48.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:47:48.454 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 6 + 1 2024-09-18T14:47:48.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=7 2024-09-18T14:47:48.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:48.456 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:47:48.456 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:47:48.456 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:47:48.457 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:48.457 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:48.457 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:48.457 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:47:48.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:48.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:47:48.540 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:48.540 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:48.540 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:47:48.540 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:48.540 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:48.549 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:48.550 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:47:48.550 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:47:48.550 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:47:48.551 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:48.551 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:48.551 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:48.551 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:47:48.552 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:48.552 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:47:48.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:48.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:48.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 7 + 4 2024-09-18T14:47:48.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=11 2024-09-18T14:47:48.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:48.650 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:47:48.650 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:47:48.650 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:47:48.651 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:48.651 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:48.651 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:48.651 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:47:48.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:48.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:47:48.733 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:48.734 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:48.734 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:47:48.734 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:48.734 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:48.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:48.744 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:47:48.744 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:47:48.744 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:47:48.744 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:48.744 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:48.744 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:48.744 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:47:48.744 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:48.744 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:47:48.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:48.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:48.839 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 11 + 4 2024-09-18T14:47:48.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=15 2024-09-18T14:47:48.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 15 = 0 ']' 2024-09-18T14:47:48.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:47:48.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 480 + 15 2024-09-18T14:47:48.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=495 2024-09-18T14:47:48.843 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:47:48.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:47:52.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:47:52.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:47:52.849 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:47:52.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:47:52.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:52.853 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:47:52.854 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:47:52.854 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:47:52.854 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:52.854 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:52.854 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:52.855 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:47:52.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:52.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:47:52.950 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:52.950 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:47:52.950 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:47:52.950 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:52.950 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:52.960 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:52.960 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:47:52.960 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:47:52.960 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:47:52.961 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:52.961 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:52.961 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:52.961 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:47:52.962 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:52.962 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:47:53.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:47:53.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:47:53.052 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 0 2024-09-18T14:47:53.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=0 2024-09-18T14:47:53.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:53.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:47:53.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:47:53.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:47:53.055 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:53.055 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:53.055 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:53.055 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:47:53.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:53.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:47:53.143 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:53.143 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:53.143 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:53.143 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:53.143 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:53.152 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:53.153 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:47:53.153 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:47:53.153 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:47:53.153 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:53.153 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:53.154 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:53.154 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:47:53.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:53.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:47:53.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:47:53.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:47:53.246 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 3 2024-09-18T14:47:53.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=3 2024-09-18T14:47:53.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:53.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:47:53.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:47:53.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:47:53.250 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:53.250 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:53.250 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:53.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:47:53.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:53.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:47:53.337 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:53.337 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:47:53.337 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:47:53.337 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:53.338 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:53.347 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:53.348 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:47:53.348 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:47:53.348 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:47:53.348 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:53.348 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:53.348 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:53.348 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:47:53.349 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:53.349 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:47:53.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:47:53.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:47:53.442 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 3 + 3 2024-09-18T14:47:53.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=6 2024-09-18T14:47:53.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:53.444 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:47:53.444 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:47:53.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:47:53.445 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:53.445 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:53.445 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:53.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:47:53.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:53.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:47:53.532 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:53.532 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:53.532 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:47:53.532 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:53.532 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:53.541 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:53.542 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:47:53.542 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:47:53.542 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:47:53.542 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:53.543 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:53.543 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:53.543 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:47:53.543 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:53.544 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:47:53.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:47:53.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:47:53.639 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 6 + 1 2024-09-18T14:47:53.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=7 2024-09-18T14:47:53.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:53.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:47:53.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:47:53.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:47:53.646 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:53.646 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:53.646 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:53.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:47:53.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:53.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:47:53.729 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:53.745 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:47:53.745 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:53.745 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:53.745 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:53.746 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:53.746 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:47:53.747 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:47:53.747 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:47:53.747 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:53.747 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:53.747 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:53.747 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:47:53.747 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:53.747 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:47:53.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:53.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:53.829 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 7 + 4 2024-09-18T14:47:53.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=11 2024-09-18T14:47:53.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:53.832 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:47:53.832 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:47:53.832 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:47:53.833 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:53.833 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:53.833 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:53.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:47:53.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:53.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:47:53.920 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:53.920 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:53.920 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:47:53.920 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:53.921 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:53.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:53.931 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:47:53.931 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:47:53.931 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:47:53.931 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:53.932 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:53.932 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:53.932 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:47:53.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:53.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:47:54.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:54.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:54.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 11 + 4 2024-09-18T14:47:54.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=15 2024-09-18T14:47:54.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 15 = 0 ']' 2024-09-18T14:47:54.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:47:54.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 495 + 15 2024-09-18T14:47:54.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=510 2024-09-18T14:47:54.034 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:47:54.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:47:58.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:47:58.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:47:58.040 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:47:58.041 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:47:58.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:58.044 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:47:58.044 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:47:58.044 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:47:58.045 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:58.045 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:58.045 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:58.046 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:47:58.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:58.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:47:58.140 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:58.140 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:47:58.140 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:47:58.140 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:58.140 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:58.150 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:58.150 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:47:58.151 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:47:58.151 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:47:58.151 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:58.151 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:58.151 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:58.152 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:47:58.152 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:58.152 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:47:58.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:47:58.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:47:58.247 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 1 2024-09-18T14:47:58.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=1 2024-09-18T14:47:58.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:58.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:47:58.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:47:58.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:47:58.250 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:58.250 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:58.250 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:58.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:47:58.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:58.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:47:58.337 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:58.337 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:58.337 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:47:58.337 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:58.337 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:58.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:58.347 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:47:58.347 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:47:58.347 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:47:58.347 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:58.347 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:58.348 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:58.348 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:47:58.348 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:58.348 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:47:58.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:47:58.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:47:58.442 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 1 + 4 2024-09-18T14:47:58.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=5 2024-09-18T14:47:58.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:58.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:47:58.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:47:58.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:47:58.446 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:58.446 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:58.446 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:58.446 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:47:58.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:58.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:47:58.527 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:58.527 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:47:58.527 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:58.528 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:58.528 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:58.537 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:58.537 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:47:58.537 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:47:58.537 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:47:58.538 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:58.538 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:58.538 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:58.538 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:47:58.538 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:58.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:47:58.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:47:58.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:47:58.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 5 + 2 2024-09-18T14:47:58.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=7 2024-09-18T14:47:58.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:58.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:47:58.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:47:58.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:47:58.631 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:58.631 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:58.631 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:58.631 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:47:58.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:58.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:47:58.712 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:58.712 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:47:58.713 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:47:58.713 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:58.713 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:58.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:58.722 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:47:58.722 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:47:58.722 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:47:58.723 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:58.723 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:58.723 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:58.724 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:47:58.724 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:58.725 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:47:58.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:47:58.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:47:58.819 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 7 + 2 2024-09-18T14:47:58.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=9 2024-09-18T14:47:58.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:58.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:47:58.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:47:58.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:47:58.822 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:58.822 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:58.823 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:58.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:47:58.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:58.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:47:58.903 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:58.903 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:58.903 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:58.903 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:58.903 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:58.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:58.913 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:47:58.914 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:47:58.914 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:47:58.914 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:58.914 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:58.914 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:58.914 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:47:58.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:58.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:47:59.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:47:59.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:47:59.008 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 9 + 3 2024-09-18T14:47:59.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=12 2024-09-18T14:47:59.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:47:59.011 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:47:59.011 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:47:59.011 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:47:59.011 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:59.012 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:59.012 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:59.012 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:47:59.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:59.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:47:59.100 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:47:59.100 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:47:59.100 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:47:59.100 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:47:59.101 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:47:59.109 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:47:59.110 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:47:59.110 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:47:59.110 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:47:59.111 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:59.111 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:59.111 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:47:59.111 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:47:59.111 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:47:59.111 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:47:59.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:47:59.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:47:59.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 12 + 3 2024-09-18T14:47:59.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=15 2024-09-18T14:47:59.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 15 = 0 ']' 2024-09-18T14:47:59.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:47:59.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 510 + 15 2024-09-18T14:47:59.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=525 2024-09-18T14:47:59.209 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:47:59.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:48:03.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:48:03.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:48:03.215 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:48:03.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:48:03.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:03.219 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:48:03.219 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:48:03.219 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:48:03.220 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:03.220 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:03.220 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:03.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:48:03.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:03.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:48:03.316 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:03.316 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:03.316 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:48:03.316 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:03.316 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:03.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:03.326 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:48:03.326 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:48:03.326 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:48:03.327 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:03.327 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:03.327 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:03.328 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:48:03.329 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:03.329 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:48:03.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:48:03.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:48:03.423 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 2 2024-09-18T14:48:03.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=2 2024-09-18T14:48:03.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:03.426 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:48:03.426 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:48:03.426 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:48:03.427 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:03.427 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:03.427 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:03.427 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:48:03.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:03.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:48:03.515 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:03.516 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:48:03.516 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:48:03.516 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:03.516 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:03.524 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:03.525 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:48:03.525 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:48:03.525 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:48:03.527 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:03.527 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:03.527 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:03.527 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:48:03.527 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:03.527 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:48:03.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:48:03.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:48:03.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 2 + 4 2024-09-18T14:48:03.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=6 2024-09-18T14:48:03.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:03.626 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:48:03.626 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:48:03.626 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:48:03.626 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:03.626 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:03.626 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:03.626 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:48:03.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:03.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:48:03.713 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:03.714 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:03.714 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:48:03.714 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:03.714 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:03.723 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:48:03.724 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:03.724 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:48:03.724 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:48:03.724 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:03.725 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:03.725 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:03.725 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:48:03.725 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:03.725 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:48:03.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:48:03.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:48:03.819 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 6 + 2 2024-09-18T14:48:03.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=8 2024-09-18T14:48:03.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:03.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:48:03.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:48:03.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:48:03.822 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:03.822 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:03.823 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:03.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:48:03.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:03.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:48:03.907 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:03.907 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:48:03.907 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:48:03.907 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:03.907 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:03.917 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:03.917 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:48:03.917 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:48:03.917 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:48:03.918 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:03.918 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:03.918 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:03.918 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:48:03.918 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:03.918 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:48:04.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:48:04.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:48:04.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 8 + 2 2024-09-18T14:48:04.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=10 2024-09-18T14:48:04.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:04.016 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:48:04.016 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:48:04.016 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:48:04.017 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:04.017 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:04.017 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:04.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:48:04.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:04.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:48:04.102 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:04.102 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:48:04.102 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:48:04.103 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:04.103 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:04.111 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:04.112 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:48:04.112 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:48:04.112 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:48:04.113 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:04.113 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:04.113 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:04.114 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:48:04.114 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:04.114 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:48:04.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:48:04.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:48:04.210 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 10 + 4 2024-09-18T14:48:04.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=14 2024-09-18T14:48:04.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:04.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:48:04.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:48:04.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:48:04.214 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:04.214 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:04.214 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:04.214 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:48:04.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:04.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:48:04.295 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:04.296 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:48:04.296 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:48:04.296 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:04.296 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:04.305 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:04.306 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:48:04.306 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:48:04.306 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:48:04.307 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:04.307 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:04.307 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:04.307 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:48:04.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:04.308 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:48:04.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:48:04.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:48:04.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 14 + 4 2024-09-18T14:48:04.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=18 2024-09-18T14:48:04.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 18 = 0 ']' 2024-09-18T14:48:04.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:48:04.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 525 + 18 2024-09-18T14:48:04.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=543 2024-09-18T14:48:04.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:48:04.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:48:08.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:48:08.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:48:08.416 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:48:08.418 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:48:08.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:08.421 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:48:08.421 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:48:08.421 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:48:08.422 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:08.422 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:08.422 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:08.422 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:48:08.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:08.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:48:08.513 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:08.513 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:08.513 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:48:08.513 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:08.513 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:08.523 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:08.523 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:48:08.523 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:48:08.524 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:48:08.524 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:08.524 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:08.524 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:08.524 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:48:08.525 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:08.525 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:48:08.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:48:08.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:48:08.620 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 2 2024-09-18T14:48:08.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=2 2024-09-18T14:48:08.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:08.622 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:48:08.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:48:08.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:48:08.623 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:08.623 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:08.624 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:08.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:48:08.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:08.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:48:08.705 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:08.705 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:48:08.705 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:48:08.705 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:08.705 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:08.714 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:08.715 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:48:08.715 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:48:08.715 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:48:08.716 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:08.716 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:08.716 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:08.716 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:48:08.717 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:08.717 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:48:08.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:48:08.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:48:08.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 2 + 4 2024-09-18T14:48:08.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=6 2024-09-18T14:48:08.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:08.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:48:08.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:48:08.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:48:08.817 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:08.817 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:08.817 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:08.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:48:08.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:08.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:48:08.906 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:08.906 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:08.906 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:48:08.906 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:08.906 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:08.916 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:08.916 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:48:08.916 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:48:08.916 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:48:08.917 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:08.917 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:08.917 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:08.917 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:48:08.917 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:08.917 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:48:09.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:48:09.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:48:09.010 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 6 + 3 2024-09-18T14:48:09.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=9 2024-09-18T14:48:09.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:09.012 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:48:09.012 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:48:09.013 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:48:09.013 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:09.013 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:09.014 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:09.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:48:09.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:09.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:48:09.099 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:09.099 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:48:09.099 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:48:09.099 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:09.099 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:09.109 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:09.110 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:48:09.110 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:48:09.110 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:48:09.110 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:09.110 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:09.110 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:09.110 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:48:09.110 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:09.111 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:48:09.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:48:09.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:48:09.199 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 9 + 2 2024-09-18T14:48:09.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=11 2024-09-18T14:48:09.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:09.201 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:48:09.201 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:48:09.201 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:48:09.201 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:09.202 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:09.202 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:09.202 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:48:09.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:09.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:48:09.288 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:09.288 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:48:09.288 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:48:09.288 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:09.288 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:09.297 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:09.297 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:48:09.297 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:48:09.297 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:48:09.298 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:09.298 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:09.298 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:09.298 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:48:09.299 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:09.299 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:48:09.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:48:09.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:48:09.395 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 11 + 4 2024-09-18T14:48:09.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=15 2024-09-18T14:48:09.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:09.397 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:48:09.397 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:48:09.397 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:48:09.398 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:09.398 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:09.398 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:09.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:48:09.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:09.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:48:09.487 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:09.488 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:48:09.488 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:48:09.488 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:09.488 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:09.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:09.497 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:48:09.497 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:48:09.498 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:48:09.498 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:09.498 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:09.498 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:09.499 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:48:09.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:09.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:48:09.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:48:09.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:48:09.595 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 15 + 3 2024-09-18T14:48:09.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=18 2024-09-18T14:48:09.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 18 = 0 ']' 2024-09-18T14:48:09.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:48:09.598 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 543 + 18 2024-09-18T14:48:09.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=561 2024-09-18T14:48:09.599 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:48:09.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:48:13.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:48:13.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:48:13.605 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:48:13.606 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:48:13.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:13.609 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:48:13.609 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:48:13.609 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:48:13.610 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:13.610 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:13.610 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:13.610 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:48:13.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:13.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:48:13.706 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:13.707 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:13.707 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:48:13.707 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:13.707 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:13.717 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:13.717 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:48:13.717 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:48:13.717 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:48:13.718 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:13.718 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:13.718 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:13.718 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:48:13.719 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:13.719 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:48:13.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:48:13.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:48:13.810 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 3 2024-09-18T14:48:13.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=3 2024-09-18T14:48:13.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:13.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:48:13.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:48:13.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:48:13.813 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:13.816 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:13.816 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:13.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:48:13.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:13.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:48:13.902 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:13.902 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:48:13.902 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:48:13.902 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:13.902 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:13.912 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:13.912 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:48:13.912 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:48:13.912 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:48:13.913 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:13.913 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:13.913 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:13.913 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:48:13.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:13.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:48:14.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:48:14.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:48:14.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 3 + 4 2024-09-18T14:48:14.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=7 2024-09-18T14:48:14.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:14.004 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:48:14.004 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:48:14.004 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:48:14.004 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:14.004 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:14.005 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:14.005 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:48:14.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:14.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:48:14.091 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:14.091 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:14.091 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 4, 2024-09-18T14:48:14.092 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:14.092 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:14.100 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:14.101 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:48:14.101 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:48:14.101 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:48:14.102 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:14.102 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:14.102 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:14.103 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:48:14.103 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:14.103 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:48:14.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:48:14.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:48:14.195 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 7 + 4 2024-09-18T14:48:14.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=11 2024-09-18T14:48:14.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:14.198 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:48:14.198 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:48:14.198 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:48:14.199 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:14.199 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:14.199 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:14.199 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:48:14.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:14.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:48:14.285 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:14.285 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:48:14.285 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:48:14.285 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:14.286 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:14.295 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:14.295 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:48:14.295 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:48:14.295 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:48:14.296 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:14.296 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:14.296 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:14.296 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:48:14.297 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:14.297 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:48:14.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:48:14.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:48:14.389 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 11 + 1 2024-09-18T14:48:14.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=12 2024-09-18T14:48:14.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:14.391 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:48:14.391 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:48:14.391 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:48:14.392 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:14.392 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:14.392 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:14.392 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:48:14.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:14.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:48:14.477 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:14.478 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:48:14.478 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:48:14.478 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:14.478 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:14.486 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:14.487 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:48:14.487 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:48:14.487 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:48:14.488 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:14.488 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:14.488 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:14.489 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:48:14.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:14.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:48:14.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:48:14.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:48:14.584 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 12 + 4 2024-09-18T14:48:14.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=16 2024-09-18T14:48:14.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:14.588 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:48:14.588 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:48:14.588 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:48:14.588 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:14.588 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:14.588 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:14.588 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:48:14.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:14.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:48:14.677 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:14.677 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:48:14.677 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:48:14.677 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:14.677 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:14.686 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:14.687 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:48:14.687 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:48:14.687 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:48:14.687 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:14.688 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:14.688 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:14.688 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:48:14.688 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:14.688 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:48:14.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:48:14.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:48:14.784 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 16 + 2 2024-09-18T14:48:14.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=18 2024-09-18T14:48:14.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 18 = 0 ']' 2024-09-18T14:48:14.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:48:14.785 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 561 + 18 2024-09-18T14:48:14.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=579 2024-09-18T14:48:14.787 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:48:14.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:48:18.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:48:18.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:48:18.793 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:48:18.794 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:48:18.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:18.797 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:48:18.797 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:48:18.797 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:48:18.798 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:18.798 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:18.798 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:18.798 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:48:18.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:18.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:48:18.895 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:18.895 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:18.895 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:48:18.895 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:18.895 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:18.904 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:18.904 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:48:18.905 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:48:18.905 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:48:18.905 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:18.905 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:18.906 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:18.906 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:48:18.906 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:18.906 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:48:18.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:48:18.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:48:18.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 3 2024-09-18T14:48:18.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=3 2024-09-18T14:48:18.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:19.000 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:48:19.001 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:48:19.001 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:48:19.001 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:19.001 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:19.001 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:19.001 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:48:19.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:19.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:48:19.084 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:19.085 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:48:19.085 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:48:19.085 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:19.085 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:19.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:19.094 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:48:19.094 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:48:19.095 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:48:19.095 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:19.095 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:19.095 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:19.096 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:48:19.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:19.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:48:19.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:48:19.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:48:19.192 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 3 + 3 2024-09-18T14:48:19.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=6 2024-09-18T14:48:19.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:19.194 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:48:19.194 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:48:19.194 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:48:19.195 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:19.195 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:19.195 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:19.196 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:48:19.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:19.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:48:19.280 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:19.509 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:19.509 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:48:19.509 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:19.509 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:19.511 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:19.512 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:48:19.512 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:48:19.512 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:48:19.512 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:19.512 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:19.512 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:19.512 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:48:19.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:19.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:48:19.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:48:19.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:48:19.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 6 + 3 2024-09-18T14:48:19.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=9 2024-09-18T14:48:19.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:19.513 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:48:19.513 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:48:19.513 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:48:19.513 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:19.513 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:19.513 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:19.513 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:48:19.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:19.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:48:19.514 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:19.514 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:48:19.514 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:48:19.514 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:19.514 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:19.515 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:19.515 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:48:19.515 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:48:19.515 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:48:19.515 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:19.515 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:19.515 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:19.516 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:48:19.516 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:19.516 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:48:19.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:48:19.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:48:19.587 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 9 + 1 2024-09-18T14:48:19.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=10 2024-09-18T14:48:19.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:19.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:48:19.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:48:19.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:48:19.591 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:19.591 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:19.591 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:19.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:48:19.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:19.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:48:19.672 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:19.672 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:48:19.672 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:48:19.672 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:19.672 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:19.681 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:19.681 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:48:19.681 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:48:19.681 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:48:19.682 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:19.682 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:19.682 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:19.683 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:48:19.683 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:19.683 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:48:19.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:48:19.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:48:19.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 10 + 4 2024-09-18T14:48:19.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=14 2024-09-18T14:48:19.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:19.774 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:48:19.774 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:48:19.774 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:48:19.775 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:19.775 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:19.775 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:19.775 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:48:19.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:19.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:48:19.862 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:19.862 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:48:19.862 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:48:19.862 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:19.862 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:19.871 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:19.872 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:48:19.872 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:48:19.872 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:48:19.873 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:19.873 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:19.873 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:19.873 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:48:19.874 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:19.874 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:48:19.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:48:19.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:48:19.971 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 14 + 1 2024-09-18T14:48:19.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=15 2024-09-18T14:48:19.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 15 = 0 ']' 2024-09-18T14:48:19.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:48:19.973 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 579 + 15 2024-09-18T14:48:19.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=594 2024-09-18T14:48:19.975 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:48:19.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:48:23.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:48:23.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:48:23.981 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:48:23.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:48:23.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:23.986 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:48:23.986 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:48:23.986 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:48:23.986 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:23.986 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:23.987 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:23.987 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:48:23.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:23.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:48:24.078 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:24.078 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:24.079 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 4, 2024-09-18T14:48:24.079 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:24.079 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:24.088 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:24.088 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:48:24.089 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:48:24.089 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:48:24.089 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:24.089 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:24.089 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:24.090 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:48:24.090 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:24.090 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:48:24.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:48:24.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:48:24.187 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 4 2024-09-18T14:48:24.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=4 2024-09-18T14:48:24.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:24.189 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:48:24.189 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:48:24.189 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:48:24.190 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:24.190 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:24.190 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:24.190 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:48:24.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:24.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:48:24.277 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:24.277 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:48:24.277 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:48:24.278 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:24.278 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:24.288 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:24.288 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:48:24.288 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:48:24.288 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:48:24.288 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:24.289 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:24.289 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:24.289 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:48:24.289 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:24.289 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:48:24.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:48:24.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:48:24.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 4 + 3 2024-09-18T14:48:24.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=7 2024-09-18T14:48:24.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:24.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:48:24.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:48:24.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:48:24.386 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:24.386 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:24.386 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:24.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:48:24.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:24.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:48:24.473 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:24.473 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:24.473 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:48:24.473 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:24.473 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:24.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:24.483 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:48:24.483 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:48:24.483 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:48:24.484 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:24.484 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:24.484 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:24.484 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:48:24.484 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:24.485 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:48:24.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:48:24.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:48:24.580 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 7 + 3 2024-09-18T14:48:24.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=10 2024-09-18T14:48:24.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:24.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:48:24.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:48:24.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:48:24.583 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:24.583 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:24.583 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:24.584 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:48:24.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:24.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:48:24.666 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:24.666 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:48:24.666 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:48:24.666 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:24.666 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:24.675 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:24.676 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:48:24.676 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:48:24.676 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:48:24.677 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:24.677 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:24.677 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:24.677 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:48:24.678 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:24.678 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:48:24.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:48:24.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:48:24.776 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 10 + 1 2024-09-18T14:48:24.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=11 2024-09-18T14:48:24.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:24.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:48:24.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:48:24.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:48:24.779 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:24.779 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:24.779 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:24.779 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:48:24.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:24.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:48:24.866 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:24.866 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:48:24.866 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:48:24.866 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:24.866 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:24.875 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:24.876 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:48:24.876 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:48:24.876 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:48:24.877 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:24.877 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:24.877 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:24.877 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:48:24.878 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:24.878 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:48:24.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:48:24.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:48:24.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 11 + 3 2024-09-18T14:48:24.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=14 2024-09-18T14:48:24.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:24.974 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:48:24.974 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:48:24.974 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:48:24.974 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:24.975 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:24.975 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:24.975 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:48:24.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:24.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:48:25.065 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:25.065 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:48:25.065 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:48:25.065 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:25.065 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:25.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:25.075 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:48:25.075 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:48:25.075 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:48:25.075 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:25.075 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:25.075 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:25.076 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:48:25.076 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:25.076 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:48:25.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:48:25.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:48:25.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 14 + 1 2024-09-18T14:48:25.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=15 2024-09-18T14:48:25.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 15 = 0 ']' 2024-09-18T14:48:25.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:48:25.166 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 594 + 15 2024-09-18T14:48:25.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=609 2024-09-18T14:48:25.169 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:48:25.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:48:29.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:48:29.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:48:29.174 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:48:29.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:48:29.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:29.179 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:48:29.179 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:48:29.179 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:48:29.179 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:29.180 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:29.180 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:29.180 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:48:29.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:29.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:48:29.273 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:29.273 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:29.273 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 4, 2024-09-18T14:48:29.273 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:29.273 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:29.283 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:29.283 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:48:29.283 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:48:29.284 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:48:29.284 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:29.284 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:29.284 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:29.285 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:48:29.285 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:29.285 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:48:29.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:48:29.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:48:29.380 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 4 2024-09-18T14:48:29.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=4 2024-09-18T14:48:29.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:29.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:48:29.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:48:29.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:48:29.383 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:29.384 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:29.384 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:29.384 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:48:29.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:29.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:48:29.470 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:29.470 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:48:29.470 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:48:29.470 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:29.470 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:29.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:29.481 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:48:29.481 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:48:29.481 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:48:29.482 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:29.482 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:29.482 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:29.482 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:48:29.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:29.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:48:29.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:48:29.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:48:29.570 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 4 + 3 2024-09-18T14:48:29.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=7 2024-09-18T14:48:29.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:29.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:48:29.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:48:29.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:48:29.574 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:29.574 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:29.574 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:29.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:48:29.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:29.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:48:29.660 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:29.660 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:29.660 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:48:29.660 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:29.660 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:29.669 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:29.670 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:48:29.670 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:48:29.670 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:48:29.670 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:29.670 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:29.671 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:29.671 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:48:29.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:29.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:48:29.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:48:29.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:48:29.768 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 7 + 3 2024-09-18T14:48:29.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=10 2024-09-18T14:48:29.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:29.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:48:29.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:48:29.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:48:29.772 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:29.772 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:29.772 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:29.772 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:48:29.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:29.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:48:29.858 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:29.858 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:48:29.858 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:48:29.859 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:29.859 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:29.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:29.868 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:48:29.868 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:48:29.868 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:48:29.869 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:29.869 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:29.869 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:29.869 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:48:29.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:29.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:48:29.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:48:29.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:48:29.965 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 10 + 1 2024-09-18T14:48:29.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=11 2024-09-18T14:48:29.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:29.967 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:48:29.967 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:48:29.967 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:48:29.967 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:29.968 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:29.968 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:29.968 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:48:29.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:29.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:48:30.054 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:30.054 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:48:30.054 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:48:30.054 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:30.054 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:30.063 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:30.064 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:48:30.064 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:48:30.064 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:48:30.065 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:30.065 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:30.065 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:30.065 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:48:30.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:30.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:48:30.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:48:30.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:48:30.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 11 + 2 2024-09-18T14:48:30.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=13 2024-09-18T14:48:30.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:30.162 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:48:30.162 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:48:30.162 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:48:30.163 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:30.163 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:30.163 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:30.163 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:48:30.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:30.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:48:30.249 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:30.250 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:30.250 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:48:30.250 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:30.250 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:30.260 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:30.260 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:48:30.260 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:48:30.260 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:48:30.261 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:30.261 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:30.261 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:30.261 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:48:30.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:30.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:48:30.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:48:30.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:48:30.353 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 13 + 0 2024-09-18T14:48:30.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=13 2024-09-18T14:48:30.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 13 = 0 ']' 2024-09-18T14:48:30.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:48:30.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 609 + 13 2024-09-18T14:48:30.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=622 2024-09-18T14:48:30.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:48:30.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:48:34.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:48:34.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:48:34.363 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:48:34.365 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:48:34.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:34.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:48:34.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:48:34.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:48:34.369 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:34.369 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:34.369 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:34.369 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:48:34.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:34.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:48:34.463 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:34.463 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:34.463 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 4, 2024-09-18T14:48:34.463 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:34.464 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:34.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:34.475 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:48:34.475 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:48:34.475 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:48:34.475 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:34.475 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:34.475 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:34.476 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:48:34.476 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:34.477 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:48:34.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:48:34.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:48:34.572 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 4 2024-09-18T14:48:34.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=4 2024-09-18T14:48:34.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:34.574 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:48:34.574 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:48:34.574 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:48:34.575 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:34.575 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:34.575 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:34.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:48:34.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:34.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:48:34.662 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:34.663 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:48:34.663 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:48:34.663 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:34.663 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:34.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:34.674 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:48:34.674 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:48:34.674 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:48:34.674 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:34.674 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:34.674 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:34.674 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:48:34.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:34.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:48:34.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:48:34.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:48:34.768 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 4 + 3 2024-09-18T14:48:34.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=7 2024-09-18T14:48:34.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:34.769 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:48:34.769 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:48:34.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:48:34.770 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:34.770 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:34.771 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:34.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:48:34.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:34.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:48:34.858 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:34.858 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:34.859 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:48:34.859 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:34.859 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:34.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:34.868 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:48:34.868 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:48:34.868 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:48:34.869 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:34.869 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:34.869 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:34.870 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:48:34.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:34.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:48:34.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:48:34.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:48:34.966 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 7 + 3 2024-09-18T14:48:34.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=10 2024-09-18T14:48:34.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:34.967 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:48:34.967 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:48:34.968 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:48:34.968 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:34.968 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:34.968 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:34.969 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:48:34.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:34.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:48:35.054 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:35.055 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:48:35.055 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:48:35.055 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:35.055 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:35.064 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:35.064 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:48:35.065 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:48:35.065 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:48:35.065 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:35.065 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:35.065 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:35.065 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:48:35.065 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:35.065 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:48:35.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:48:35.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:48:35.159 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 10 + 2 2024-09-18T14:48:35.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=12 2024-09-18T14:48:35.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:35.162 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:48:35.162 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:48:35.162 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:48:35.163 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:35.163 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:35.163 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:35.163 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:48:35.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:35.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:48:35.244 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:35.244 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:48:35.244 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:48:35.244 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:35.244 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:35.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:35.254 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:48:35.254 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:48:35.254 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:48:35.254 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:35.255 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:35.255 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:35.255 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:48:35.255 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:35.256 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:48:35.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:48:35.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:48:35.351 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 12 + 3 2024-09-18T14:48:35.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=15 2024-09-18T14:48:35.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:35.353 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:48:35.353 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:48:35.353 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:48:35.354 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:35.354 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:35.354 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:35.354 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:48:35.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:35.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:48:35.441 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:35.441 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:35.441 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:48:35.441 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:35.441 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:35.450 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:35.451 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:48:35.451 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:48:35.451 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:48:35.452 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:35.452 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:35.452 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:35.453 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:48:35.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:35.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:48:35.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:48:35.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:48:35.546 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 15 + 0 2024-09-18T14:48:35.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=15 2024-09-18T14:48:35.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 15 = 0 ']' 2024-09-18T14:48:35.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:48:35.549 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 622 + 15 2024-09-18T14:48:35.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=637 2024-09-18T14:48:35.551 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:48:35.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:48:39.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:48:39.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:48:39.557 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:48:39.559 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:48:39.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:39.562 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:48:39.562 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:48:39.562 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:48:39.562 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:39.563 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:39.563 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:39.563 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:48:39.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:39.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:48:39.658 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:39.659 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:39.659 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:48:39.659 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:39.659 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:39.669 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:39.670 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:48:39.670 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:48:39.670 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:48:39.670 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:39.671 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:39.671 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:39.671 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:48:39.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:39.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:48:39.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:48:39.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:48:39.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 4 2024-09-18T14:48:39.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=4 2024-09-18T14:48:39.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:39.767 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:48:39.767 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:48:39.767 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:48:39.768 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:39.768 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:39.768 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:39.768 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:48:39.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:39.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:48:39.848 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:39.849 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:48:39.849 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:48:39.849 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:39.849 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:39.858 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:39.859 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:48:39.859 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:48:39.859 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:48:39.859 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:39.859 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:39.859 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:39.860 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:48:39.861 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:39.861 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:48:39.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:48:39.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:48:39.952 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 4 + 1 2024-09-18T14:48:39.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=5 2024-09-18T14:48:39.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:39.954 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:48:39.954 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:48:39.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:48:39.955 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:39.955 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:39.955 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:39.956 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:48:39.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:39.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:48:40.042 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:40.042 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:40.042 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:48:40.042 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:40.042 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:40.051 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:40.052 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:48:40.052 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:48:40.052 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:48:40.052 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:40.053 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:40.053 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:40.053 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:48:40.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:40.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:48:40.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:48:40.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:48:40.150 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 5 + 1 2024-09-18T14:48:40.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=6 2024-09-18T14:48:40.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:40.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:48:40.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:48:40.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:48:40.154 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:40.154 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:40.154 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:40.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:48:40.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:40.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:48:40.239 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:40.240 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 3, 2024-09-18T14:48:40.240 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:48:40.240 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:40.240 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:40.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:40.249 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:48:40.249 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:48:40.249 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:48:40.250 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:40.250 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:40.250 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:40.250 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:48:40.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:40.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:48:40.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:48:40.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:48:40.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 6 + 3 2024-09-18T14:48:40.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=9 2024-09-18T14:48:40.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:40.348 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:48:40.348 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:48:40.348 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:48:40.349 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:40.349 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:40.349 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:40.349 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:48:40.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:40.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:48:40.436 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:40.436 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:40.436 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:48:40.436 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:40.436 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:40.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:40.446 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:48:40.446 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:48:40.446 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:48:40.447 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:40.447 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:40.447 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:40.447 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:48:40.448 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:40.448 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:48:40.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:48:40.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:48:40.542 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 9 + 3 2024-09-18T14:48:40.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=12 2024-09-18T14:48:40.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:40.545 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:48:40.545 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:48:40.545 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:48:40.545 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:40.545 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:40.545 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:40.546 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:48:40.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:40.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:48:40.631 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:40.631 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:40.631 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:48:40.632 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:40.632 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:40.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:40.641 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:48:40.641 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:48:40.641 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:48:40.642 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:40.642 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:40.642 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:40.643 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:48:40.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:40.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:48:40.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:48:40.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:48:40.738 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 12 + 0 2024-09-18T14:48:40.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=12 2024-09-18T14:48:40.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 12 = 0 ']' 2024-09-18T14:48:40.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:48:40.740 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 637 + 12 2024-09-18T14:48:40.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=649 2024-09-18T14:48:40.742 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:48:40.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:48:44.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:48:44.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:48:44.747 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:48:44.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:48:44.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:44.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:48:44.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:48:44.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:48:44.753 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:44.753 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:44.753 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:44.753 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:48:44.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:44.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:48:44.849 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:44.849 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:44.849 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 4, 2024-09-18T14:48:44.850 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:44.850 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:44.860 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:48:44.860 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:48:44.860 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:44.860 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:48:44.861 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:44.861 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:44.861 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:44.861 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:48:44.862 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:44.862 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:48:44.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=4 2024-09-18T14:48:44.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2024-09-18T14:48:44.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 4 2024-09-18T14:48:44.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=4 2024-09-18T14:48:44.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:44.954 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:48:44.954 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:48:44.954 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:48:44.955 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:44.955 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:44.955 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:44.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:48:44.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:44.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:48:45.041 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:45.041 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:48:45.041 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:48:45.041 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:45.041 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:45.051 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:45.051 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:48:45.051 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:48:45.052 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:48:45.052 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:45.052 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:45.052 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:45.053 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:48:45.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:45.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:48:45.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:48:45.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:48:45.149 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 4 + 1 2024-09-18T14:48:45.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=5 2024-09-18T14:48:45.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:45.152 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:48:45.152 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:48:45.152 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:48:45.153 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:45.153 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:45.153 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:45.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:48:45.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:45.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:48:45.239 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:45.240 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:45.240 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:48:45.240 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:45.240 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:45.248 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:45.249 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:48:45.249 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:48:45.249 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:48:45.249 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:45.250 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:45.250 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:45.250 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:48:45.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:45.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:48:45.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:48:45.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:48:45.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 5 + 1 2024-09-18T14:48:45.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=6 2024-09-18T14:48:45.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:45.349 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:48:45.349 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:48:45.349 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:48:45.349 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:45.350 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:45.350 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:45.350 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:48:45.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:45.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:48:45.436 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:45.436 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 3, 2024-09-18T14:48:45.436 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:48:45.436 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:45.437 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:45.446 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:45.446 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:48:45.446 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:48:45.446 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:48:45.447 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:45.447 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:45.447 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:45.448 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:48:45.448 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:45.448 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:48:45.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:48:45.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:48:45.544 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 6 + 3 2024-09-18T14:48:45.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=9 2024-09-18T14:48:45.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:45.546 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:48:45.546 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:48:45.546 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:48:45.547 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:45.547 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:45.547 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:45.547 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:48:45.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:45.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:48:45.632 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:45.633 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:45.633 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:48:45.633 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:45.633 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:45.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:45.642 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:48:45.642 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:48:45.642 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:48:45.644 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:45.644 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:45.644 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:45.644 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:48:45.645 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:45.645 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:48:45.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:48:45.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:48:45.738 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 9 + 3 2024-09-18T14:48:45.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=12 2024-09-18T14:48:45.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:45.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:48:45.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:48:45.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:48:45.741 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:45.742 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:45.742 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:45.742 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:48:45.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:45.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:48:45.829 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:45.829 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:45.829 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:48:45.829 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:45.829 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:45.838 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:45.839 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:48:45.839 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:48:45.839 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:48:45.840 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:45.840 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:45.840 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:45.840 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:48:45.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:45.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:48:45.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:48:45.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:48:45.938 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 12 + 0 2024-09-18T14:48:45.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=12 2024-09-18T14:48:45.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 12 = 0 ']' 2024-09-18T14:48:45.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:48:45.940 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 649 + 12 2024-09-18T14:48:45.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=661 2024-09-18T14:48:45.941 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:48:45.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:48:49.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:48:49.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:48:49.948 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:48:49.949 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:48:49.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:49.952 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:48:49.952 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:48:49.952 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:48:49.953 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:49.953 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:49.953 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:49.953 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:48:49.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:49.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:48:50.048 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:50.048 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:50.048 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:48:50.048 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:50.048 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:50.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:50.059 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:48:50.059 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:48:50.059 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:48:50.059 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:50.059 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:50.059 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:50.060 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:48:50.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:50.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:48:50.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:48:50.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:48:50.170 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 3 2024-09-18T14:48:50.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=3 2024-09-18T14:48:50.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:50.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:48:50.172 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:48:50.172 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:48:50.172 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:50.172 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:50.173 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:50.173 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:48:50.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:50.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:48:50.259 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:50.259 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:48:50.259 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:48:50.259 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:50.259 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:50.269 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:50.269 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:48:50.269 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:48:50.269 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:48:50.269 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:50.269 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:50.269 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:50.269 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:48:50.270 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:50.270 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:48:50.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:48:50.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:48:50.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 3 + 1 2024-09-18T14:48:50.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=4 2024-09-18T14:48:50.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:50.365 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:48:50.365 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:48:50.366 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:48:50.366 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:50.366 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:50.366 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:50.367 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:48:50.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:50.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:48:50.447 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:50.447 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:50.447 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:48:50.447 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:50.447 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:50.457 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:50.458 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:48:50.458 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:48:50.458 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:48:50.459 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:50.459 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:50.459 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:50.459 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:48:50.460 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:50.460 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:48:50.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:48:50.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:48:50.555 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 4 + 1 2024-09-18T14:48:50.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=5 2024-09-18T14:48:50.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:50.557 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:48:50.557 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:48:50.557 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:48:50.558 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:50.558 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:50.558 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:50.558 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:48:50.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:50.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:48:50.645 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:50.645 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:48:50.645 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:48:50.645 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:50.645 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:50.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:50.655 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:48:50.655 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:48:50.656 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:48:50.657 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:50.657 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:50.657 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:50.657 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:48:50.657 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:50.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:48:50.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:48:50.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:48:50.746 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 5 + 3 2024-09-18T14:48:50.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=8 2024-09-18T14:48:50.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:50.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:48:50.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:48:50.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:48:50.750 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:50.750 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:50.750 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:50.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:48:50.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:50.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:48:50.832 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:50.832 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:50.832 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:48:50.832 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:50.832 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:50.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:50.841 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:48:50.841 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:48:50.841 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:48:50.842 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:50.842 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:50.842 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:50.843 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:48:50.843 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:50.843 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:48:50.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:48:50.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:48:50.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 8 + 2 2024-09-18T14:48:50.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=10 2024-09-18T14:48:50.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:50.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:48:50.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:48:50.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:48:50.936 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:50.936 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:50.936 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:50.936 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:48:50.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:50.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:48:51.025 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:51.025 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:51.025 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:48:51.025 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:51.025 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:51.034 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:51.035 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:48:51.035 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:48:51.035 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:48:51.036 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:51.036 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:51.036 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:51.036 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:48:51.037 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:51.037 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:48:51.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:48:51.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:48:51.130 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 10 + 0 2024-09-18T14:48:51.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=10 2024-09-18T14:48:51.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 10 = 0 ']' 2024-09-18T14:48:51.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:48:51.132 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 661 + 10 2024-09-18T14:48:51.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=671 2024-09-18T14:48:51.133 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:48:51.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:48:55.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:48:55.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:48:55.140 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:48:55.141 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:48:55.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:55.144 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:48:55.144 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:48:55.144 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:48:55.145 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:55.145 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:55.145 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:55.146 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:48:55.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:55.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:48:55.238 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:55.238 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:55.238 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:48:55.238 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:55.238 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:55.247 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:55.248 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:48:55.248 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:48:55.248 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:48:55.249 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:55.249 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:55.249 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:55.249 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:48:55.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:55.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:48:55.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:48:55.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:48:55.343 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 3 2024-09-18T14:48:55.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=3 2024-09-18T14:48:55.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:55.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:48:55.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:48:55.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:48:55.347 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:55.347 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:55.347 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:55.347 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:48:55.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:55.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:48:55.432 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:55.433 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:48:55.433 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:48:55.433 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:55.433 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:55.442 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:55.442 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:48:55.443 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:48:55.443 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:48:55.443 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:55.443 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:55.443 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:55.444 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:48:55.444 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:55.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:48:55.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:48:55.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:48:55.536 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 3 + 1 2024-09-18T14:48:55.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=4 2024-09-18T14:48:55.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:55.538 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:48:55.538 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:48:55.538 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:48:55.538 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:55.539 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:55.539 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:55.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:48:55.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:55.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:48:55.625 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:55.625 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:55.625 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:48:55.625 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:55.625 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:55.635 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:55.635 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:48:55.635 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:48:55.635 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:48:55.636 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:55.636 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:55.636 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:55.636 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:48:55.637 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:55.637 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:48:55.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:48:55.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:48:55.731 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 4 + 0 2024-09-18T14:48:55.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=4 2024-09-18T14:48:55.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:55.734 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:48:55.734 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:48:55.734 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:48:55.735 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:55.735 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:55.735 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:55.735 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:48:55.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:55.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:48:55.816 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:55.816 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:48:55.816 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:48:55.816 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:55.816 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:55.825 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:55.826 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:48:55.826 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:48:55.826 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:48:55.827 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:55.827 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:55.827 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:55.827 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:48:55.828 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:55.828 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:48:55.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:48:55.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:48:55.918 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 4 + 3 2024-09-18T14:48:55.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=7 2024-09-18T14:48:55.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:55.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:48:55.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:48:55.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:48:55.921 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:55.921 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:55.921 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:55.921 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:48:55.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:55.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:48:56.008 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:56.009 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:56.009 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:48:56.009 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:56.009 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:56.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:56.018 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:48:56.019 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:48:56.019 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:48:56.019 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:56.019 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:56.019 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:56.020 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:48:56.021 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:56.021 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:48:56.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:48:56.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:48:56.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 7 + 2 2024-09-18T14:48:56.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=9 2024-09-18T14:48:56.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:48:56.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:48:56.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:48:56.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:48:56.121 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:56.121 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:56.122 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:56.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:48:56.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:56.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:48:56.206 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:48:56.206 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:48:56.206 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:48:56.206 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:48:56.206 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:48:56.214 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:48:56.215 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:48:56.215 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:48:56.215 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:48:56.216 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:48:56.216 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:56.216 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:48:56.216 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:48:56.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:48:56.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:48:56.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:48:56.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:48:56.313 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 9 + 1 2024-09-18T14:48:56.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=10 2024-09-18T14:48:56.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 10 = 0 ']' 2024-09-18T14:48:56.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:48:56.316 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 671 + 10 2024-09-18T14:48:56.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=681 2024-09-18T14:48:56.317 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:48:56.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:49:00.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:49:00.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:49:00.323 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:49:00.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:49:00.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:00.328 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:49:00.328 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:49:00.328 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:49:00.329 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:00.329 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:00.329 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:00.329 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:49:00.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:00.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:49:00.421 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:00.421 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:00.421 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:49:00.421 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:00.421 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:00.431 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:00.431 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:49:00.431 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:49:00.431 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:49:00.432 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:00.432 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:00.432 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:00.432 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:49:00.433 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:00.433 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:49:00.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:49:00.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:49:00.530 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 3 2024-09-18T14:49:00.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=3 2024-09-18T14:49:00.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:00.533 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:49:00.533 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:49:00.533 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:49:00.534 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:00.534 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:00.534 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:00.534 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:49:00.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:00.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:49:00.614 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:00.614 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:49:00.614 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:00.614 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:00.614 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:00.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:00.624 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:49:00.624 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:49:00.624 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:49:00.625 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:00.625 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:00.625 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:00.625 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:49:00.625 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:00.625 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:49:00.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:49:00.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:49:00.719 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 3 + 2 2024-09-18T14:49:00.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=5 2024-09-18T14:49:00.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:00.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:49:00.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:49:00.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:49:00.723 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:00.723 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:00.723 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:00.723 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:49:00.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:00.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:49:00.804 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:00.805 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:00.805 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:00.805 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:00.805 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:00.815 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:00.815 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:49:00.815 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:49:00.815 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:49:00.816 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:00.816 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:00.816 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:00.816 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:49:00.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:00.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:49:00.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:00.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:00.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 5 + 0 2024-09-18T14:49:00.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=5 2024-09-18T14:49:00.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:00.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:49:00.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:49:00.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:49:00.914 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:00.914 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:00.914 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:00.914 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:49:00.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:00.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:49:01.000 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:01.000 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:49:01.001 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:49:01.001 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:01.001 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:01.010 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:01.010 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:49:01.011 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:49:01.011 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:49:01.011 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:01.011 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:01.011 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:01.012 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:49:01.012 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:01.013 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:49:01.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:49:01.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:49:01.109 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 5 + 2 2024-09-18T14:49:01.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=7 2024-09-18T14:49:01.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:01.112 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:49:01.112 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:49:01.112 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:49:01.113 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:01.113 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:01.113 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:01.113 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:49:01.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:01.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:49:01.198 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:01.198 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:01.198 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:49:01.198 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:01.198 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:01.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:01.208 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:49:01.208 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:49:01.208 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:49:01.209 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:01.209 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:01.209 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:01.209 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:49:01.210 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:01.210 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:49:01.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:49:01.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:49:01.303 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 7 + 1 2024-09-18T14:49:01.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=8 2024-09-18T14:49:01.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:01.305 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:49:01.305 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:49:01.306 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:49:01.307 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:01.307 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:01.307 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:01.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:49:01.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:01.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:49:01.392 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:01.392 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:01.392 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:49:01.392 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:01.392 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:01.401 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:01.402 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:49:01.402 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:49:01.402 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:49:01.402 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:01.403 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:01.403 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:01.403 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:49:01.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:01.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:49:01.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:49:01.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:49:01.498 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 8 + 1 2024-09-18T14:49:01.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=9 2024-09-18T14:49:01.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 9 = 0 ']' 2024-09-18T14:49:01.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:49:01.501 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 681 + 9 2024-09-18T14:49:01.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=690 2024-09-18T14:49:01.503 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:49:01.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:49:05.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:49:05.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:49:05.509 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:49:05.511 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:49:05.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:05.513 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:49:05.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:49:05.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:49:05.514 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:05.515 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:05.515 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:05.515 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:49:05.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:05.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:49:05.605 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:05.606 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:05.606 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:49:05.606 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:05.606 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:05.615 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:05.616 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:49:05.616 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:49:05.616 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:49:05.617 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:05.617 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:05.617 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:05.617 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:49:05.618 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:05.618 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:49:05.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:49:05.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:49:05.715 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 3 2024-09-18T14:49:05.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=3 2024-09-18T14:49:05.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:05.718 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:49:05.718 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:49:05.718 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:49:05.718 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:05.719 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:05.719 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:05.719 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:49:05.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:05.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:49:05.800 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:05.800 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:49:05.801 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:05.801 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:05.801 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:05.810 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:05.811 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:49:05.811 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:49:05.811 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:49:05.811 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:05.811 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:05.812 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:05.812 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:49:05.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:05.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:49:05.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:49:05.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:49:05.906 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 3 + 2 2024-09-18T14:49:05.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=5 2024-09-18T14:49:05.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:05.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:49:05.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:49:05.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:49:05.908 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:05.908 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:05.908 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:05.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:49:05.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:05.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:49:05.988 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:05.988 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:05.988 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:05.988 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:05.988 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:05.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:05.997 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:49:05.997 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:49:05.997 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:49:05.997 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:05.997 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:05.998 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:05.998 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:49:05.999 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:05.999 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:49:06.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:06.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:06.092 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 5 + 0 2024-09-18T14:49:06.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=5 2024-09-18T14:49:06.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:06.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:49:06.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:49:06.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:49:06.095 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:06.095 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:06.095 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:06.095 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:49:06.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:06.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:49:06.178 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:06.178 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:06.178 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:49:06.178 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:06.178 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:06.189 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:06.189 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:49:06.189 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:49:06.189 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:49:06.190 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:06.190 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:06.190 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:06.190 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:49:06.190 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:06.190 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:49:06.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:49:06.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:49:06.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 5 + 1 2024-09-18T14:49:06.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=6 2024-09-18T14:49:06.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:06.282 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:49:06.282 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:49:06.282 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:49:06.283 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:06.283 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:06.283 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:06.283 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:49:06.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:06.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:49:06.368 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:06.368 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:06.369 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:06.369 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:06.369 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:06.378 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:06.378 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:49:06.378 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:49:06.378 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:49:06.379 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:06.379 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:06.379 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:06.380 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:49:06.380 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:06.380 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:49:06.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:06.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:06.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 6 + 0 2024-09-18T14:49:06.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=6 2024-09-18T14:49:06.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:06.478 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:49:06.478 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:49:06.478 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:49:06.479 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:06.480 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:06.480 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:06.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:49:06.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:06.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:49:06.562 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:06.563 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:06.563 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:49:06.563 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:06.563 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:06.572 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:06.572 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:49:06.573 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:49:06.573 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:49:06.573 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:06.573 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:06.573 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:06.574 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:49:06.574 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:06.574 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:49:06.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:49:06.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:49:06.670 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 6 + 1 2024-09-18T14:49:06.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=7 2024-09-18T14:49:06.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 7 = 0 ']' 2024-09-18T14:49:06.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:49:06.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 690 + 7 2024-09-18T14:49:06.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=697 2024-09-18T14:49:06.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:49:06.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:49:10.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:49:10.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:49:10.680 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:49:10.681 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:49:10.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:10.684 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:49:10.684 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:49:10.684 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:49:10.685 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:10.685 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:10.685 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:10.685 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:49:10.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:10.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:49:10.778 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:10.778 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:10.778 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 3, 2024-09-18T14:49:10.778 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:10.778 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:10.787 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:10.788 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:49:10.789 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:49:10.789 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:49:10.790 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:10.790 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:10.790 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:10.790 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:49:10.791 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:10.791 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:49:10.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:49:10.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:49:10.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 3 2024-09-18T14:49:10.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=3 2024-09-18T14:49:10.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:10.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:49:10.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:49:10.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:49:10.886 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:10.886 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:10.887 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:10.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:49:10.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:10.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:49:10.973 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:10.973 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 3, 2024-09-18T14:49:10.973 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:10.973 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:10.973 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:10.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:10.983 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:49:10.983 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:49:10.983 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:49:10.984 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:10.984 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:10.984 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:10.984 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:49:10.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:10.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:49:11.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=3 2024-09-18T14:49:11.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2024-09-18T14:49:11.079 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 3 + 3 2024-09-18T14:49:11.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=6 2024-09-18T14:49:11.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:11.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:49:11.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:49:11.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:49:11.082 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:11.082 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:11.082 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:11.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:49:11.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:11.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:49:11.171 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:11.171 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:11.171 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:49:11.171 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:11.171 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:11.180 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:11.181 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:49:11.181 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:49:11.181 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:49:11.181 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:11.181 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:11.182 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:11.182 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:49:11.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:11.183 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:49:11.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:49:11.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:49:11.277 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 6 + 1 2024-09-18T14:49:11.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=7 2024-09-18T14:49:11.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:11.278 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:49:11.278 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:49:11.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:49:11.279 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:11.279 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:11.279 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:11.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:49:11.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:11.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:49:11.366 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:11.366 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:11.366 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:49:11.366 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:11.366 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:11.375 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:11.376 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:49:11.376 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:49:11.376 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:49:11.376 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:11.377 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:11.377 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:11.377 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:49:11.377 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:11.377 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:49:11.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:49:11.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:49:11.471 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 7 + 1 2024-09-18T14:49:11.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=8 2024-09-18T14:49:11.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:11.473 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:49:11.473 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:49:11.473 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:49:11.473 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:11.474 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:11.474 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:11.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:49:11.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:11.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:49:11.561 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:11.561 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:11.561 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:11.561 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:11.561 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:11.571 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:11.571 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:49:11.571 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:49:11.571 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:49:11.572 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:11.572 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:11.572 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:11.572 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:49:11.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:11.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:49:11.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:11.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:11.669 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 8 + 0 2024-09-18T14:49:11.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=8 2024-09-18T14:49:11.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:11.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:49:11.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:49:11.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:49:11.672 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:11.673 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:11.673 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:11.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:49:11.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:11.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:49:11.761 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:11.761 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:11.761 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:49:11.761 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:11.762 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:11.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:11.771 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:49:11.771 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:49:11.771 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:49:11.772 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:11.772 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:11.772 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:11.772 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:49:11.772 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:11.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:49:11.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:49:11.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:49:11.861 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 8 + 1 2024-09-18T14:49:11.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=9 2024-09-18T14:49:11.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 9 = 0 ']' 2024-09-18T14:49:11.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:49:11.864 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 697 + 9 2024-09-18T14:49:11.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=706 2024-09-18T14:49:11.866 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:49:11.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:49:15.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:49:15.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:49:15.871 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:49:15.873 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:49:15.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:15.876 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:49:15.876 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:49:15.876 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:49:15.877 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:15.877 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:15.877 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:15.877 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:49:15.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:15.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:49:15.970 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:15.970 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:15.970 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:49:15.970 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:15.970 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:15.979 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:15.980 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:49:15.980 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:49:15.980 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:49:15.981 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:15.981 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:15.981 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:15.982 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:49:15.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:15.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:49:16.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:49:16.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:49:16.078 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 2 2024-09-18T14:49:16.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=2 2024-09-18T14:49:16.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:16.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:49:16.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:49:16.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:49:16.081 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:16.081 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:16.081 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:16.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:49:16.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:16.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:49:16.168 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:16.168 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:49:16.168 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:16.168 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:16.168 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:16.177 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:16.178 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:49:16.178 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:49:16.178 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:49:16.179 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:16.179 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:16.179 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:16.179 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:49:16.180 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:16.180 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:49:16.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:49:16.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:49:16.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 2 + 2 2024-09-18T14:49:16.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=4 2024-09-18T14:49:16.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:16.277 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:49:16.277 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:49:16.277 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:49:16.278 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:16.278 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:16.278 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:16.278 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:49:16.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:16.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:49:16.363 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:16.363 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:16.363 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:49:16.363 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:16.363 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:16.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:16.373 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:49:16.373 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:49:16.373 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:49:16.373 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:16.373 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:16.373 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:16.374 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:49:16.374 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:16.374 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:49:16.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:49:16.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:49:16.469 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 4 + 1 2024-09-18T14:49:16.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=5 2024-09-18T14:49:16.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:16.471 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:49:16.472 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:49:16.472 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:49:16.472 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:16.472 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:16.472 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:16.473 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:49:16.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:16.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:49:16.558 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:16.558 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:16.559 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:16.559 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:16.559 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:16.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:16.568 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:49:16.568 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:49:16.568 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:49:16.569 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:16.569 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:16.569 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:16.570 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:49:16.570 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:16.570 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:49:16.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:16.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:16.667 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 5 + 0 2024-09-18T14:49:16.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=5 2024-09-18T14:49:16.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:16.670 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:49:16.670 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:49:16.671 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:49:16.671 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:16.671 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:16.671 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:16.671 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:49:16.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:16.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:49:16.754 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:16.754 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:16.754 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:16.754 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:16.754 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:16.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:16.764 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:49:16.764 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:49:16.764 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:49:16.765 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:16.765 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:16.765 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:16.765 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:49:16.766 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:16.766 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:49:16.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:16.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:16.855 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 5 + 0 2024-09-18T14:49:16.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=5 2024-09-18T14:49:16.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:16.858 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:49:16.858 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:49:16.858 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:49:16.858 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:16.859 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:16.859 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:16.859 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:49:16.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:16.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:49:16.940 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:16.940 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:16.940 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:49:16.941 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:16.941 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:16.950 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:16.950 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:49:16.951 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:49:16.951 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:49:16.952 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:16.952 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:16.952 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:16.952 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:49:16.953 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:16.953 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:49:17.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:49:17.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:49:17.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 5 + 1 2024-09-18T14:49:17.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=6 2024-09-18T14:49:17.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 6 = 0 ']' 2024-09-18T14:49:17.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:49:17.051 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 706 + 6 2024-09-18T14:49:17.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=712 2024-09-18T14:49:17.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:49:17.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:49:21.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:49:21.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:49:21.059 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:49:21.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:49:21.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:21.063 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:49:21.063 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:49:21.064 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:49:21.064 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:21.064 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:21.064 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:21.065 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:49:21.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:21.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:49:21.159 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:21.160 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:21.160 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 2, 2024-09-18T14:49:21.160 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:21.160 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:21.170 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:21.171 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:49:21.171 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:49:21.171 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:49:21.172 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:21.172 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:21.172 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:21.172 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:49:21.173 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:21.173 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:49:21.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:49:21.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:49:21.264 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 2 2024-09-18T14:49:21.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=2 2024-09-18T14:49:21.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:21.267 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:49:21.267 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:49:21.267 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:49:21.267 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:21.268 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:21.268 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:21.268 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:49:21.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:21.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:49:21.357 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:21.357 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 2, 2024-09-18T14:49:21.357 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:21.357 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:21.357 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:21.366 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:21.367 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:49:21.367 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:49:21.367 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:49:21.368 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:21.368 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:21.368 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:21.368 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:49:21.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:21.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:49:21.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=2 2024-09-18T14:49:21.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2024-09-18T14:49:21.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 2 + 2 2024-09-18T14:49:21.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=4 2024-09-18T14:49:21.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:21.468 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:49:21.468 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:49:21.468 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:49:21.468 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:21.469 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:21.469 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:21.469 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:49:21.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:21.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:49:21.555 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:21.555 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:21.555 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:49:21.555 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:21.556 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:21.566 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:21.566 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:49:21.566 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:49:21.566 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:49:21.566 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:21.566 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:21.566 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:21.566 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:49:21.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:21.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:49:21.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:49:21.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:49:21.659 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 4 + 1 2024-09-18T14:49:21.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=5 2024-09-18T14:49:21.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:21.662 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:49:21.662 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:49:21.662 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:49:21.662 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:21.662 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:21.662 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:21.663 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:49:21.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:21.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:49:21.750 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:21.750 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:21.750 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:21.751 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:21.751 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:21.760 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:21.760 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:49:21.760 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:49:21.760 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:49:21.761 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:21.761 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:21.761 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:21.761 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:49:21.762 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:21.762 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:49:21.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:21.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:21.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 5 + 0 2024-09-18T14:49:21.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=5 2024-09-18T14:49:21.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:21.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:49:21.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:49:21.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:49:21.853 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:21.853 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:21.853 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:21.853 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:49:21.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:21.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:49:21.940 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:21.940 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:21.940 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:21.940 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:21.940 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:21.949 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:21.950 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:49:21.950 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:49:21.950 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:49:21.950 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:21.950 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:21.951 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:21.951 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:49:21.952 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:21.952 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:49:22.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:22.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:22.044 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 5 + 0 2024-09-18T14:49:22.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=5 2024-09-18T14:49:22.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:22.046 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:49:22.046 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:49:22.046 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:49:22.047 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:22.047 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:22.047 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:22.047 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:49:22.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:22.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:49:22.135 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:22.135 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:22.135 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:22.136 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:22.136 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:22.144 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:22.145 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:49:22.145 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:49:22.145 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:49:22.146 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:22.146 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:22.146 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:22.146 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:49:22.147 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:22.147 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:49:22.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:22.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:22.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 5 + 0 2024-09-18T14:49:22.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=5 2024-09-18T14:49:22.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 5 = 0 ']' 2024-09-18T14:49:22.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:49:22.244 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 712 + 5 2024-09-18T14:49:22.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=717 2024-09-18T14:49:22.246 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:49:22.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:49:26.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:49:26.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:49:26.253 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:49:26.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:49:26.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:26.256 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:49:26.256 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:49:26.256 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:49:26.257 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:26.257 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:26.257 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:26.257 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:49:26.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:26.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:49:26.349 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:26.349 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:26.349 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:49:26.349 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:26.350 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:26.359 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:26.359 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:49:26.360 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:49:26.360 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:49:26.360 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:26.360 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:26.361 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:26.361 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:49:26.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:26.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:49:26.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:49:26.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:49:26.455 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 1 2024-09-18T14:49:26.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=1 2024-09-18T14:49:26.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:26.458 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:49:26.458 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:49:26.458 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:49:26.459 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:26.459 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:26.459 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:26.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:49:26.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:26.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:49:26.545 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:26.545 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 1, 2024-09-18T14:49:26.545 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:26.545 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:26.545 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:26.555 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:26.555 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:49:26.555 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:49:26.556 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:49:26.556 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:26.557 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:26.557 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:26.557 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:49:26.557 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:26.557 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:49:26.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:49:26.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:49:26.653 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 1 + 1 2024-09-18T14:49:26.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=2 2024-09-18T14:49:26.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:26.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:49:26.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:49:26.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:49:26.656 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:26.656 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:26.656 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:26.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:49:26.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:26.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:49:26.743 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:26.743 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:26.744 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:49:26.744 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:26.744 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:26.753 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:26.753 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:49:26.753 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:49:26.753 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:49:26.754 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:26.754 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:26.754 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:26.754 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:49:26.754 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:26.754 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:49:26.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:49:26.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:49:26.843 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 2 + 1 2024-09-18T14:49:26.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=3 2024-09-18T14:49:26.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:26.845 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:49:26.845 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:49:26.845 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:49:26.846 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:26.846 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:26.846 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:26.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:49:26.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:26.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:49:26.934 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:26.934 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:26.934 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:26.934 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:26.934 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:26.943 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:26.943 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:49:26.944 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:49:26.944 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:49:26.944 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:26.944 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:26.945 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:26.945 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:49:26.945 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:26.945 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:49:27.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:27.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:27.039 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 3 + 0 2024-09-18T14:49:27.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=3 2024-09-18T14:49:27.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:27.041 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:49:27.041 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:49:27.041 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:49:27.042 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:27.042 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:27.042 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:27.042 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:49:27.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:27.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:49:27.130 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:27.131 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:27.131 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:27.131 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:27.131 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:27.140 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:27.141 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:49:27.141 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:49:27.141 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:49:27.141 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:27.141 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:27.141 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:27.141 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:49:27.141 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:27.142 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:49:27.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:27.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:27.236 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 3 + 0 2024-09-18T14:49:27.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=3 2024-09-18T14:49:27.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:27.239 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:49:27.239 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:49:27.239 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:49:27.239 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:27.239 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:27.239 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:27.240 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:49:27.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:27.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:49:27.324 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:27.324 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:27.324 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:27.325 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:27.325 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:27.334 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:27.335 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:49:27.335 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:49:27.335 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:49:27.335 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:27.335 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:27.335 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:27.336 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:49:27.336 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:27.336 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:49:27.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:27.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:27.431 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 3 + 0 2024-09-18T14:49:27.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=3 2024-09-18T14:49:27.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 3 = 0 ']' 2024-09-18T14:49:27.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:49:27.433 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 717 + 3 2024-09-18T14:49:27.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=720 2024-09-18T14:49:27.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:49:27.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:49:31.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:49:31.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:49:31.440 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:49:31.441 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:49:31.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:31.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:49:31.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:49:31.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:49:31.446 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:31.446 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:31.446 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:31.447 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:49:31.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:31.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:49:31.538 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:31.538 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:31.538 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 1, 2024-09-18T14:49:31.538 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:31.538 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:31.547 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:31.548 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:49:31.548 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:49:31.548 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:49:31.549 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:31.549 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:31.549 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:31.549 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:49:31.550 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:31.550 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:49:31.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=1 2024-09-18T14:49:31.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2024-09-18T14:49:31.642 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 1 2024-09-18T14:49:31.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=1 2024-09-18T14:49:31.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:31.645 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:49:31.645 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:49:31.645 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:49:31.645 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:31.646 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:31.646 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:31.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:49:31.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:31.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:49:31.733 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:31.734 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:31.734 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:31.734 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:31.734 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:31.744 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:31.744 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:49:31.744 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:49:31.744 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:49:31.745 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:31.745 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:31.745 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:31.745 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:49:31.745 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:31.745 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:49:31.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:31.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:31.839 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 1 + 0 2024-09-18T14:49:31.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=1 2024-09-18T14:49:31.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:31.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:49:31.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:49:31.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:49:31.842 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:31.842 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:31.842 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:31.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:49:31.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:31.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:49:31.922 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:31.922 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:31.922 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:31.922 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:31.922 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:31.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:31.932 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:49:31.932 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:49:31.932 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:49:31.932 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:31.932 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:31.932 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:31.932 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:49:31.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:31.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:49:32.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:32.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:32.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 1 + 0 2024-09-18T14:49:32.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=1 2024-09-18T14:49:32.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:32.028 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:49:32.028 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:49:32.028 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:49:32.028 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:32.029 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:32.029 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:32.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:49:32.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:32.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:49:32.116 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:32.116 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:32.117 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:32.117 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:32.117 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:32.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:32.126 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:49:32.126 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:49:32.126 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:49:32.127 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:32.127 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:32.127 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:32.127 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:49:32.127 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:32.127 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:49:32.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:32.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:32.223 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 1 + 0 2024-09-18T14:49:32.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=1 2024-09-18T14:49:32.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:32.225 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:49:32.226 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:49:32.226 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:49:32.226 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:32.226 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:32.226 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:32.226 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:49:32.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:32.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:49:32.311 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:32.312 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:32.312 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:32.312 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:32.312 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:32.321 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:32.322 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:49:32.322 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:49:32.322 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:49:32.322 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:32.322 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:32.323 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:32.323 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:49:32.323 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:32.324 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:49:32.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:32.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:32.419 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 1 + 0 2024-09-18T14:49:32.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=1 2024-09-18T14:49:32.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:32.422 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:49:32.422 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:49:32.422 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:49:32.423 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:32.423 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:32.423 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:32.423 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:49:32.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:32.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:49:32.507 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:32.507 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:32.507 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:32.507 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:32.508 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:32.517 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:32.517 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:49:32.517 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:49:32.517 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:49:32.518 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:32.518 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:32.518 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:32.518 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:49:32.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:32.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:49:32.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:32.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:32.607 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 1 + 0 2024-09-18T14:49:32.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=1 2024-09-18T14:49:32.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 1 = 0 ']' 2024-09-18T14:49:32.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2024-09-18T14:49:32.609 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 720 + 1 2024-09-18T14:49:32.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=721 2024-09-18T14:49:32.611 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:49:32.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:49:36.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:49:36.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:49:36.617 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:49:36.618 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:49:36.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:36.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:49:36.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:49:36.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:49:36.622 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:36.622 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:36.622 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:36.622 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:49:36.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:36.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:49:36.717 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:36.717 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:36.717 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:36.717 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:36.717 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:36.727 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:36.727 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:49:36.727 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:49:36.727 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:49:36.728 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:36.728 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:36.728 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:36.728 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:49:36.729 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:36.729 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:49:36.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:36.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:36.826 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 0 2024-09-18T14:49:36.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=0 2024-09-18T14:49:36.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:36.828 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:49:36.828 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:49:36.829 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:49:36.829 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:36.829 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:36.829 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:36.830 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:49:36.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:36.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:49:36.917 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:36.917 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:36.917 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:36.917 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:36.917 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:36.926 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:36.927 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:49:36.927 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:49:36.927 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:49:36.928 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:36.928 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:36.928 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:36.928 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:49:36.928 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:36.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:49:37.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:37.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:37.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 0 2024-09-18T14:49:37.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=0 2024-09-18T14:49:37.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:37.025 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:49:37.025 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:49:37.025 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:49:37.025 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:37.025 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:37.025 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:37.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:49:37.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:37.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:49:37.108 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:37.108 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:37.108 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:37.108 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:37.109 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:37.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:37.119 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:49:37.119 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:49:37.119 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:49:37.119 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:37.119 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:37.119 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:37.120 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:49:37.120 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:37.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:49:37.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:37.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:37.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 0 2024-09-18T14:49:37.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=0 2024-09-18T14:49:37.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:37.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:49:37.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:49:37.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:49:37.218 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:37.218 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:37.219 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:37.219 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:49:37.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:37.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:49:37.307 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:37.307 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:37.307 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:37.307 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:37.307 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:37.316 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:37.317 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:49:37.317 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:49:37.317 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:49:37.318 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:37.318 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:37.318 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:37.318 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:49:37.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:37.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:49:37.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:37.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:37.412 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 0 2024-09-18T14:49:37.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=0 2024-09-18T14:49:37.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:37.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:49:37.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:49:37.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:49:37.414 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:37.414 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:37.415 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:37.415 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:49:37.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:37.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:49:37.495 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:37.495 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:37.495 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:37.495 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:37.495 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:37.504 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:37.504 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:49:37.505 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:49:37.505 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:49:37.505 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:37.505 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:37.505 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:37.506 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:49:37.506 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:37.506 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:49:37.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:37.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:37.604 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 0 2024-09-18T14:49:37.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=0 2024-09-18T14:49:37.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:37.606 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:49:37.606 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:49:37.607 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:49:37.607 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:37.608 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:37.608 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:37.608 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:49:37.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:37.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:49:37.689 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:37.689 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:37.689 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:37.689 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:37.689 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:37.699 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:37.700 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:49:37.700 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:49:37.700 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:49:37.701 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:37.701 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:37.701 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:37.701 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:49:37.702 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:37.702 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:49:37.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:37.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:37.796 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 0 2024-09-18T14:49:37.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=0 2024-09-18T14:49:37.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 0 = 0 ']' 2024-09-18T14:49:37.798 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:135: TEST_recover_unexpected: expr 0 + 1 2024-09-18T14:49:37.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:135: TEST_recover_unexpected: zerocount=1 2024-09-18T14:49:37.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 1 -gt 3 ']' 2024-09-18T14:49:37.800 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 721 + 0 2024-09-18T14:49:37.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=721 2024-09-18T14:49:37.802 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:49:37.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:49:41.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:49:41.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:49:41.808 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:49:41.809 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:49:41.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:41.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:49:41.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:49:41.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:49:41.813 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:41.813 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:41.813 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:41.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:49:41.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:41.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:49:41.906 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:41.907 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:41.907 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:41.907 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:41.907 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:41.917 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:41.918 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:49:41.918 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:49:41.918 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:49:41.919 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:41.919 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:41.919 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:41.919 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:49:41.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:41.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:49:42.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:42.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:42.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 0 2024-09-18T14:49:42.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=0 2024-09-18T14:49:42.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:42.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:49:42.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:49:42.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:49:42.017 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:42.018 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:42.018 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:42.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:49:42.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:42.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:49:42.105 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:42.105 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:42.105 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:42.105 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:42.105 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:42.114 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:42.115 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:49:42.116 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:49:42.116 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:49:42.116 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:42.116 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:42.116 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:42.116 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:49:42.116 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:42.117 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:49:42.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:42.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:42.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 0 2024-09-18T14:49:42.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=0 2024-09-18T14:49:42.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:42.214 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:49:42.214 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:49:42.215 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:49:42.215 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:42.215 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:42.215 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:42.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:49:42.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:42.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:49:42.300 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:42.300 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:42.300 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:42.301 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:42.301 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:42.310 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:42.310 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:49:42.310 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:49:42.310 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:49:42.311 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:42.311 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:42.311 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:42.312 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:49:42.313 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:42.313 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:49:42.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:42.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:42.406 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 0 2024-09-18T14:49:42.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=0 2024-09-18T14:49:42.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:42.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:49:42.409 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:49:42.409 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:49:42.409 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:42.409 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:42.409 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:42.409 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:49:42.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:42.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:49:42.495 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:42.495 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:42.496 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:42.496 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:42.496 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:42.505 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:42.505 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:49:42.505 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:49:42.505 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:49:42.506 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:42.506 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:42.506 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:42.506 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:49:42.507 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:42.507 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:49:42.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:42.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:42.598 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 0 2024-09-18T14:49:42.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=0 2024-09-18T14:49:42.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:42.600 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:49:42.601 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:49:42.601 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:49:42.601 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:42.601 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:42.601 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:42.602 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:49:42.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:42.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:49:42.687 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:42.687 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:42.687 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:42.687 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:42.687 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:42.695 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:42.696 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:49:42.696 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:49:42.696 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:49:42.697 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:42.697 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:42.697 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:42.698 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:49:42.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:42.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:49:42.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:42.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:42.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 0 2024-09-18T14:49:42.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=0 2024-09-18T14:49:42.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:42.794 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:49:42.794 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:49:42.794 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:49:42.795 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:42.795 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:42.795 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:42.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:49:42.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:42.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:49:42.879 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:42.879 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:42.879 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:42.879 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:42.879 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:42.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:42.890 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:49:42.890 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:49:42.890 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:49:42.891 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:42.891 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:42.891 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:42.891 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:49:42.892 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:42.892 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:49:42.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:42.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:42.988 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 0 2024-09-18T14:49:42.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=0 2024-09-18T14:49:42.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 0 = 0 ']' 2024-09-18T14:49:42.990 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:135: TEST_recover_unexpected: expr 1 + 1 2024-09-18T14:49:42.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:135: TEST_recover_unexpected: zerocount=2 2024-09-18T14:49:42.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 2 -gt 3 ']' 2024-09-18T14:49:42.992 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 721 + 0 2024-09-18T14:49:42.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=721 2024-09-18T14:49:42.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:49:42.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:49:46.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:49:46.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:49:47.000 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:49:47.001 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:49:47.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:47.004 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:49:47.004 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:49:47.004 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:49:47.005 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:47.005 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:47.005 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:47.006 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:49:47.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:47.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:49:47.099 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:47.099 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:47.099 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:47.099 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:47.099 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:47.109 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:47.109 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:49:47.109 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:49:47.109 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:49:47.110 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:47.110 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:47.110 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:47.110 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:49:47.111 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:47.111 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:49:47.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:47.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:47.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 0 2024-09-18T14:49:47.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=0 2024-09-18T14:49:47.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:47.210 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:49:47.211 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:49:47.211 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:49:47.211 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:47.211 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:47.211 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:47.211 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:49:47.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:47.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:49:47.297 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:47.297 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:47.297 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:47.297 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:47.298 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:47.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:47.308 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:49:47.308 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:49:47.308 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:49:47.308 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:47.308 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:47.308 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:47.309 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:49:47.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:47.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:49:47.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:47.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:47.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 0 2024-09-18T14:49:47.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=0 2024-09-18T14:49:47.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:47.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:49:47.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:49:47.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:49:47.407 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:47.407 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:47.407 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:47.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:49:47.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:47.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:49:47.488 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:47.488 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:47.488 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:47.488 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:47.488 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:47.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:47.497 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:49:47.497 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:49:47.497 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:49:47.498 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:47.498 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:47.498 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:47.499 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:49:47.499 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:47.499 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:49:47.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:47.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:47.595 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 0 2024-09-18T14:49:47.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=0 2024-09-18T14:49:47.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:47.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:49:47.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:49:47.598 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:49:47.598 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:47.598 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:47.598 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:47.599 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:49:47.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:47.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:49:47.679 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:47.679 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:47.679 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:47.679 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:47.679 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:47.689 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:47.689 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:49:47.689 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:49:47.689 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:49:47.690 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:47.690 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:47.690 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:47.690 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:49:47.690 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:47.690 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:49:47.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:47.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:47.779 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 0 2024-09-18T14:49:47.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=0 2024-09-18T14:49:47.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:47.781 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:49:47.782 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:49:47.782 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:49:47.782 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:47.782 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:47.782 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:47.782 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:49:47.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:47.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:49:47.864 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:47.864 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:47.864 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:47.864 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:47.864 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:47.873 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:47.874 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:49:47.874 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:49:47.874 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:49:47.875 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:47.875 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:47.875 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:47.875 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:49:47.876 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:47.876 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:49:47.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:47.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:47.973 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 0 2024-09-18T14:49:47.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=0 2024-09-18T14:49:47.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:47.975 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:49:47.975 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:49:47.975 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:49:47.976 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:47.976 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:47.976 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:47.977 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:49:47.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:47.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:49:48.063 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:48.064 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:48.064 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:48.064 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:48.064 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:48.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:48.073 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:49:48.073 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:49:48.073 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:49:48.073 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:48.074 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:48.074 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:48.074 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:49:48.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:48.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:49:48.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:48.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:48.167 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 0 2024-09-18T14:49:48.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=0 2024-09-18T14:49:48.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 0 = 0 ']' 2024-09-18T14:49:48.169 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:135: TEST_recover_unexpected: expr 2 + 1 2024-09-18T14:49:48.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:135: TEST_recover_unexpected: zerocount=3 2024-09-18T14:49:48.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 3 -gt 3 ']' 2024-09-18T14:49:48.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 721 + 0 2024-09-18T14:49:48.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: total=721 2024-09-18T14:49:48.174 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: expr 2 '*' 2 2024-09-18T14:49:48.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:141: TEST_recover_unexpected: sleep 4 2024-09-18T14:49:52.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:121: TEST_recover_unexpected: true 2024-09-18T14:49:52.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: pass=0 2024-09-18T14:49:52.180 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: expr 6 - 1 2024-09-18T14:49:52.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: seq 0 5 2024-09-18T14:49:52.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:52.184 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:49:52.184 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:49:52.184 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:49:52.185 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:52.185 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:52.185 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:52.186 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:49:52.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:52.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:49:52.277 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:52.277 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:52.277 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:52.278 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:52.278 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:52.287 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:52.288 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.0 2024-09-18T14:49:52.288 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:49:52.288 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:49:52.289 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:52.289 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:52.289 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:52.289 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.0.asok 2024-09-18T14:49:52.289 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:52.290 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.0.asok dump_scrub_reservations 2024-09-18T14:49:52.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:52.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:52.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 0 2024-09-18T14:49:52.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=0 2024-09-18T14:49:52.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:52.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:49:52.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:49:52.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:49:52.387 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:52.387 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:52.387 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:52.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:49:52.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:52.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:49:52.473 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:52.473 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:52.473 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:52.473 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:52.473 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:52.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:52.483 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.1 2024-09-18T14:49:52.483 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:49:52.483 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:49:52.483 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:52.484 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:52.484 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:52.484 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.1.asok 2024-09-18T14:49:52.484 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:52.484 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.1.asok dump_scrub_reservations 2024-09-18T14:49:52.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:52.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:52.578 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 0 2024-09-18T14:49:52.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=0 2024-09-18T14:49:52.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:52.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:49:52.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:49:52.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:49:52.581 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:52.582 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:52.582 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:52.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:49:52.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:52.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:49:52.661 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:52.661 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:52.662 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:52.662 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:52.662 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:52.670 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:52.671 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.2 2024-09-18T14:49:52.671 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:49:52.671 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:49:52.672 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:52.672 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:52.672 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:52.672 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.2.asok 2024-09-18T14:49:52.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:52.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.2.asok dump_scrub_reservations 2024-09-18T14:49:52.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:52.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:52.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 0 2024-09-18T14:49:52.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=0 2024-09-18T14:49:52.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:52.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:49:52.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:49:52.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:49:52.774 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:52.774 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:52.774 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:52.774 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:49:52.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:52.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:49:52.862 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:52.863 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:52.863 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:52.863 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:52.863 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:52.872 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:52.872 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.3 2024-09-18T14:49:52.873 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T14:49:52.873 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T14:49:52.873 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:52.873 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:52.874 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:52.874 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.3.asok 2024-09-18T14:49:52.874 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:52.874 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.3.asok dump_scrub_reservations 2024-09-18T14:49:52.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:52.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:52.970 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 0 2024-09-18T14:49:52.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=0 2024-09-18T14:49:52.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:52.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:49:52.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:49:52.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:49:52.973 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:52.973 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:52.973 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:52.973 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:49:52.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:52.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:49:53.061 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:53.062 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:53.062 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:53.062 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:53.062 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:53.071 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:53.072 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.4 2024-09-18T14:49:53.072 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2024-09-18T14:49:53.072 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2024-09-18T14:49:53.073 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:53.073 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:53.073 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:53.073 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.4.asok 2024-09-18T14:49:53.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:53.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.4.asok dump_scrub_reservations 2024-09-18T14:49:53.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:53.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:53.166 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 0 2024-09-18T14:49:53.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=0 2024-09-18T14:49:53.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:124: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:53.169 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:49:53.169 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:49:53.169 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:49:53.170 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:53.170 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:53.170 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:53.170 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:49:53.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:53.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:49:53.252 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:49:53.252 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_local": 0, 2024-09-18T14:49:53.252 INFO:tasks.workunit.client.0.smithi046.stdout: "scrubs_remote": 0, 2024-09-18T14:49:53.252 INFO:tasks.workunit.client.0.smithi046.stdout: "osd_max_scrubs": 4 2024-09-18T14:49:53.252 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:49:53.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2024-09-18T14:49:53.261 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: get_asok_path osd.5 2024-09-18T14:49:53.262 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T14:49:53.262 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T14:49:53.262 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:53.262 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:53.262 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:49:53.263 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61461/ceph-osd.5.asok 2024-09-18T14:49:53.264 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: CEPH_ARGS= 2024-09-18T14:49:53.264 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.61461/ceph-osd.5.asok dump_scrub_reservations 2024-09-18T14:49:53.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: scrubs=0 2024-09-18T14:49:53.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:128: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2024-09-18T14:49:53.354 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: expr 0 + 0 2024-09-18T14:49:53.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:132: TEST_recover_unexpected: pass=0 2024-09-18T14:49:53.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: '[' 0 = 0 ']' 2024-09-18T14:49:53.356 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:135: TEST_recover_unexpected: expr 3 + 1 2024-09-18T14:49:53.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:135: TEST_recover_unexpected: zerocount=4 2024-09-18T14:49:53.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: '[' 4 -gt 3 ']' 2024-09-18T14:49:53.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:138: TEST_recover_unexpected: break 2024-09-18T14:49:53.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:145: TEST_recover_unexpected: seq 0 5 2024-09-18T14:49:53.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:145: TEST_recover_unexpected: for i in $(seq 0 5) 2024-09-18T14:49:53.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:147: TEST_recover_unexpected: ceph pg dump pgs 2024-09-18T14:49:53.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:147: TEST_recover_unexpected: grep +scrubbing 2024-09-18T14:49:53.669 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:49:53.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:151: TEST_recover_unexpected: sleep 2 2024-09-18T14:49:55.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:145: TEST_recover_unexpected: for i in $(seq 0 5) 2024-09-18T14:49:55.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:147: TEST_recover_unexpected: ceph pg dump pgs 2024-09-18T14:49:55.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:147: TEST_recover_unexpected: grep +scrubbing 2024-09-18T14:49:56.002 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:49:56.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:151: TEST_recover_unexpected: sleep 2 2024-09-18T14:49:58.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:145: TEST_recover_unexpected: for i in $(seq 0 5) 2024-09-18T14:49:58.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:147: TEST_recover_unexpected: ceph pg dump pgs 2024-09-18T14:49:58.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:147: TEST_recover_unexpected: grep +scrubbing 2024-09-18T14:49:58.333 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:49:58.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:151: TEST_recover_unexpected: sleep 2 2024-09-18T14:50:00.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:145: TEST_recover_unexpected: for i in $(seq 0 5) 2024-09-18T14:50:00.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:147: TEST_recover_unexpected: ceph pg dump pgs 2024-09-18T14:50:00.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:147: TEST_recover_unexpected: grep +scrubbing 2024-09-18T14:50:00.665 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:50:00.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:151: TEST_recover_unexpected: sleep 2 2024-09-18T14:50:02.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:145: TEST_recover_unexpected: for i in $(seq 0 5) 2024-09-18T14:50:02.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:147: TEST_recover_unexpected: ceph pg dump pgs 2024-09-18T14:50:02.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:147: TEST_recover_unexpected: grep +scrubbing 2024-09-18T14:50:02.998 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:50:03.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:151: TEST_recover_unexpected: sleep 2 2024-09-18T14:50:05.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:145: TEST_recover_unexpected: for i in $(seq 0 5) 2024-09-18T14:50:05.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:147: TEST_recover_unexpected: ceph pg dump pgs 2024-09-18T14:50:05.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:147: TEST_recover_unexpected: grep +scrubbing 2024-09-18T14:50:05.328 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:50:05.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:151: TEST_recover_unexpected: sleep 2 2024-09-18T14:50:07.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:154: TEST_recover_unexpected: echo 721 total reservations seen 2024-09-18T14:50:07.339 INFO:tasks.workunit.client.0.smithi046.stdout:721 total reservations seen 2024-09-18T14:50:07.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:159: TEST_recover_unexpected: expr 16 '*' 3 '*' 3 2024-09-18T14:50:07.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:159: TEST_recover_unexpected: actual_reservations=144 2024-09-18T14:50:07.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:160: TEST_recover_unexpected: '[' 721 -lt 144 ']' 2024-09-18T14:50:07.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:165: TEST_recover_unexpected: return 0 2024-09-18T14:50:07.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:47: run: teardown td/osd-scrub-dump 2024-09-18T14:50:07.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-dump 2024-09-18T14:50:07.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:50:07.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-dump KILL 2024-09-18T14:50:07.343 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:50:07.343 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:50:07.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:50:07.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:50:07.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:50:07.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:50:07.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:50:07.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:50:07.469 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:50:07.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:50:07.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:50:07.471 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:50:07.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:50:07.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:50:07.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:50:07.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:50:07.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:50:07.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:50:07.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-dump 2024-09-18T14:50:07.608 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:50:07.608 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:50:07.608 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:50:07.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61461 2024-09-18T14:50:07.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:50:07.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:50:07.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T14:50:07.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-scrub-dump 0 2024-09-18T14:50:07.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-dump 2024-09-18T14:50:07.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T14:50:07.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-dump KILL 2024-09-18T14:50:07.611 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:50:07.611 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:50:07.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:50:07.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:50:07.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:50:07.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:50:07.615 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:50:07.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:50:07.617 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:50:07.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:50:07.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:50:07.619 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:50:07.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:50:07.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:50:07.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:50:07.622 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:50:07.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:50:07.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T14:50:07.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-dump 2024-09-18T14:50:07.629 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:50:07.629 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:50:07.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61461 2024-09-18T14:50:07.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61461 2024-09-18T14:50:07.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:50:07.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:50:07.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T14:50:07.633 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T14:50:07.634 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T14:50:07.650 INFO:tasks.workunit:Running workunit scrub/osd-scrub-repair.sh... 2024-09-18T14:50:07.650 DEBUG:teuthology.orchestra.run.smithi046:workunit test scrub/osd-scrub-repair.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh 2024-09-18T14:50:07.705 INFO:tasks.workunit.client.0.smithi046.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-09-18T14:50:07.706 INFO:tasks.workunit.client.0.smithi046.stderr:++ TIMEOUT=300 2024-09-18T14:50:07.706 INFO:tasks.workunit.client.0.smithi046.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-09-18T14:50:07.706 INFO:tasks.workunit.client.0.smithi046.stderr:++ MAX_TIMEOUT=15 2024-09-18T14:50:07.706 INFO:tasks.workunit.client.0.smithi046.stderr:++ PG_NUM=4 2024-09-18T14:50:07.706 INFO:tasks.workunit.client.0.smithi046.stderr:++ TMPDIR=/tmp 2024-09-18T14:50:07.706 INFO:tasks.workunit.client.0.smithi046.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-09-18T14:50:07.706 INFO:tasks.workunit.client.0.smithi046.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-09-18T14:50:07.707 INFO:tasks.workunit.client.0.smithi046.stderr:++ type xmlstarlet 2024-09-18T14:50:07.707 INFO:tasks.workunit.client.0.smithi046.stderr:++ XMLSTARLET=xmlstarlet 2024-09-18T14:50:07.707 INFO:tasks.workunit.client.0.smithi046.stderr:+++ uname 2024-09-18T14:50:07.707 INFO:tasks.workunit.client.0.smithi046.stderr:++ '[' Linux = FreeBSD ']' 2024-09-18T14:50:07.708 INFO:tasks.workunit.client.0.smithi046.stderr:++ SED=sed 2024-09-18T14:50:07.708 INFO:tasks.workunit.client.0.smithi046.stderr:++ AWK=awk 2024-09-18T14:50:07.709 INFO:tasks.workunit.client.0.smithi046.stderr:+++ stty -a 2024-09-18T14:50:07.709 INFO:tasks.workunit.client.0.smithi046.stderr:+++ head -1 2024-09-18T14:50:07.709 INFO:tasks.workunit.client.0.smithi046.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-09-18T14:50:07.710 INFO:tasks.workunit.client.0.smithi046.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T14:50:07.712 INFO:tasks.workunit.client.0.smithi046.stderr:++ termwidth= 2024-09-18T14:50:07.712 INFO:tasks.workunit.client.0.smithi046.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-09-18T14:50:07.712 INFO:tasks.workunit.client.0.smithi046.stderr:++ DIFFCOLOPTS='-y ' 2024-09-18T14:50:07.713 INFO:tasks.workunit.client.0.smithi046.stderr:++ KERNCORE=kernel.core_pattern 2024-09-18T14:50:07.713 INFO:tasks.workunit.client.0.smithi046.stderr:++ EXTRA_OPTS= 2024-09-18T14:50:07.724 INFO:tasks.workunit.client.0.smithi046.stderr:++ test '' = TESTS 2024-09-18T14:50:07.725 INFO:tasks.workunit.client.0.smithi046.stderr:++ uname 2024-09-18T14:50:07.726 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' Linux = FreeBSD ']' 2024-09-18T14:50:07.726 INFO:tasks.workunit.client.0.smithi046.stderr:+ use_ec_overwrite=true 2024-09-18T14:50:07.727 INFO:tasks.workunit.client.0.smithi046.stderr:+ getjson=no 2024-09-18T14:50:07.727 INFO:tasks.workunit.client.0.smithi046.stderr:+ jqfilter='def walk(f): 2024-09-18T14:50:07.727 INFO:tasks.workunit.client.0.smithi046.stderr: . as $in 2024-09-18T14:50:07.727 INFO:tasks.workunit.client.0.smithi046.stderr: | if type == "object" then 2024-09-18T14:50:07.727 INFO:tasks.workunit.client.0.smithi046.stderr: reduce keys[] as $key 2024-09-18T14:50:07.727 INFO:tasks.workunit.client.0.smithi046.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-18T14:50:07.727 INFO:tasks.workunit.client.0.smithi046.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-18T14:50:07.727 INFO:tasks.workunit.client.0.smithi046.stderr: else f 2024-09-18T14:50:07.727 INFO:tasks.workunit.client.0.smithi046.stderr: end; 2024-09-18T14:50:07.727 INFO:tasks.workunit.client.0.smithi046.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-18T14:50:07.727 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-18T14:50:07.727 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-18T14:50:07.727 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-18T14:50:07.727 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-18T14:50:07.728 INFO:tasks.workunit.client.0.smithi046.stderr:+ sortkeys='import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-18T14:50:07.734 INFO:tasks.workunit.client.0.smithi046.stderr:+ main osd-scrub-repair 2024-09-18T14:50:07.734 INFO:tasks.workunit.client.0.smithi046.stderr:+ local dir=td/osd-scrub-repair 2024-09-18T14:50:07.734 INFO:tasks.workunit.client.0.smithi046.stderr:+ shift 2024-09-18T14:50:07.734 INFO:tasks.workunit.client.0.smithi046.stderr:+ shopt -s -o xtrace 2024-09-18T14:50:07.734 INFO:tasks.workunit.client.0.smithi046.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T14:50:07.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-18T14:50:07.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-18T14:50:07.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T14:50:07.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T14:50:07.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T14:50:07.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T14:50:07.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T14:50:07.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T14:50:07.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-scrub-repair 2024-09-18T14:50:07.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:51: run: local dir=td/osd-scrub-repair 2024-09-18T14:50:07.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:52: run: shift 2024-09-18T14:50:07.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:54: run: export CEPH_MON=127.0.0.1:7107 2024-09-18T14:50:07.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:54: run: CEPH_MON=127.0.0.1:7107 2024-09-18T14:50:07.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: export CEPH_ARGS 2024-09-18T14:50:07.736 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:56: run: uuidgen 2024-09-18T14:50:07.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:56: run: CEPH_ARGS+='--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none ' 2024-09-18T14:50:07.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: CEPH_ARGS+='--mon-host=127.0.0.1:7107 ' 2024-09-18T14:50:07.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:58: run: CEPH_ARGS+='--osd-skip-data-digest=false ' 2024-09-18T14:50:07.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:60: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-09-18T14:50:07.737 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: set 2024-09-18T14:50:07.737 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T14:50:07.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: local 'funcs=TEST_allow_repair_during_recovery 2024-09-18T14:50:07.746 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_auto_repair_bluestore_basic 2024-09-18T14:50:07.746 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_auto_repair_bluestore_failed 2024-09-18T14:50:07.746 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_auto_repair_bluestore_failed_norecov 2024-09-18T14:50:07.746 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_auto_repair_bluestore_scrub 2024-09-18T14:50:07.746 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_auto_repair_bluestore_tag 2024-09-18T14:50:07.746 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_auto_repair_erasure_coded_appends 2024-09-18T14:50:07.746 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_auto_repair_erasure_coded_overwrites 2024-09-18T14:50:07.746 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_corrupt_and_repair_jerasure_appends 2024-09-18T14:50:07.747 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_corrupt_and_repair_jerasure_overwrites 2024-09-18T14:50:07.747 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_corrupt_and_repair_lrc_appends 2024-09-18T14:50:07.747 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_corrupt_and_repair_lrc_overwrites 2024-09-18T14:50:07.747 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_corrupt_and_repair_replicated 2024-09-18T14:50:07.747 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_corrupt_scrub_erasure_appends 2024-09-18T14:50:07.747 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_corrupt_scrub_erasure_overwrites 2024-09-18T14:50:07.747 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_corrupt_scrub_replicated 2024-09-18T14:50:07.747 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_corrupt_snapset_scrub_rep 2024-09-18T14:50:07.747 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_list_missing_erasure_coded_appends 2024-09-18T14:50:07.747 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_list_missing_erasure_coded_overwrites 2024-09-18T14:50:07.747 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_periodic_scrub_replicated 2024-09-18T14:50:07.748 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_repair_stats 2024-09-18T14:50:07.748 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_repair_stats_ec 2024-09-18T14:50:07.748 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_request_scrub_priority 2024-09-18T14:50:07.748 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_scrub_warning 2024-09-18T14:50:07.748 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_skip_non_repair_during_recovery 2024-09-18T14:50:07.748 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_unfound_erasure_coded_appends 2024-09-18T14:50:07.748 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_unfound_erasure_coded_overwrites' 2024-09-18T14:50:07.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T14:50:07.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T14:50:07.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T14:50:07.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T14:50:07.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T14:50:07.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:50:07.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T14:50:07.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:50:07.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:50:07.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:50:07.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:50:07.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:50:07.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:50:07.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:50:07.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:50:07.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:50:07.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:50:07.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:50:07.755 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:50:07.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:50:07.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:50:07.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:50:07.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:50:07.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:50:07.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:50:07.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T14:50:07.763 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:50:07.763 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:50:07.763 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:50:07.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T14:50:07.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:50:07.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:50:07.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T14:50:07.767 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:50:07.767 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:50:07.767 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:50:07.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T14:50:07.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:50:07.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T14:50:07.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T14:50:07.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:50:07.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T14:50:07.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_allow_repair_during_recovery td/osd-scrub-repair 2024-09-18T14:50:07.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:113: TEST_allow_repair_during_recovery: local dir=td/osd-scrub-repair 2024-09-18T14:50:07.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:114: TEST_allow_repair_during_recovery: local poolname=rbd 2024-09-18T14:50:07.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:116: TEST_allow_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-09-18T14:50:07.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T14:50:07.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:50:07.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:50:07.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:50:07.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T14:50:07.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-09-18T14:50:08.151 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:50:08.151 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:50:08.151 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:50:08.152 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:50:08.153 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:50:08.153 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:50:08.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:50:08.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-18T14:50:08.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:50:08.391 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:50:08.391 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:50:08.391 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:50:08.391 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:50:08.392 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:50:08.393 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:50:08.393 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:50:08.393 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:50:08.394 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:50:08.394 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:50:08.394 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:50:08.395 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T14:50:08.395 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:50:08.395 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T14:50:08.498 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:50:08.498 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:50:08.498 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:50:08.498 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:50:08.498 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:50:08.498 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:50:08.499 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:50:08.499 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:50:08.499 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:50:08.499 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:50:08.499 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:50:08.500 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T14:50:08.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:50:08.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T14:50:08.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:117: TEST_allow_repair_during_recovery: run_mgr td/osd-scrub-repair x 2024-09-18T14:50:08.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T14:50:08.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:50:08.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:50:08.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:50:08.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T14:50:08.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:50:09.460 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:50:09.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:50:09.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:50:09.461 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:50:09.461 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:50:09.462 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:50:09.462 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:50:09.462 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:50:09.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:50:09.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:118: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-18T14:50:09.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T14:50:09.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:50:09.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:50:09.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:50:09.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T14:50:09.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T14:50:09.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:50:09.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:50:09.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:50:09.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T14:50:09.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T14:50:09.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:50:09.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:50:09.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T14:50:09.503 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:50:09.503 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:50:09.503 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:50:09.505 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:50:09.505 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:50:09.505 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:50:09.505 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:50:09.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:50:09.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:50:09.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:50:09.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:50:09.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T14:50:09.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T14:50:09.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:50:09.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:50:09.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:50:09.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:50:09.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:50:09.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-09-18T14:50:09.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T14:50:09.507 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:50:09.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9a72ec9c-4c15-4e49-8d80-69184823502e 2024-09-18T14:50:09.508 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 9a72ec9c-4c15-4e49-8d80-69184823502e 2024-09-18T14:50:09.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9a72ec9c-4c15-4e49-8d80-69184823502e' 2024-09-18T14:50:09.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:50:09.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCh6OpmWlNCHxAAJ0/PfYMzRvuMu2lsU/6hBQ== 2024-09-18T14:50:09.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCh6OpmWlNCHxAAJ0/PfYMzRvuMu2lsU/6hBQ=="}' 2024-09-18T14:50:09.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9a72ec9c-4c15-4e49-8d80-69184823502e -i td/osd-scrub-repair/0/new.json 2024-09-18T14:50:09.808 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:50:09.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-18T14:50:09.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQCh6OpmWlNCHxAAJ0/PfYMzRvuMu2lsU/6hBQ== --osd-uuid 9a72ec9c-4c15-4e49-8d80-69184823502e 2024-09-18T14:50:09.855 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:50:09.850+0000 7fbd83d83540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:50:09.863 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:50:09.858+0000 7fbd83d83540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:50:09.865 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:50:09.858+0000 7fbd83d83540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:50:09.866 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:50:09.858+0000 7fbd83d83540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T14:50:09.866 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:50:09.858+0000 7fbd83d83540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T14:50:09.866 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:50:09.858+0000 7fbd83d83540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T14:50:09.966 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:50:09.958+0000 7fbd83d83540 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-18T14:50:16.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T14:50:16.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:50:16.236 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T14:50:16.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:50:16.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:50:16.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:50:16.622 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T14:50:16.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-18T14:50:16.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:50:16.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:50:16.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:50:16.655 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:50:16.646+0000 7f2ca0238540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:50:16.658 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:50:16.650+0000 7f2ca0238540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:50:16.659 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:50:16.650+0000 7f2ca0238540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:50:16.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:50:16.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:50:16.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:50:16.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:50:16.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:50:16.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:16.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:50:16.924 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:50:16.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:16.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:50:17.202 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:50:17.194+0000 7f2ca0238540 -1 Falling back to public interface 2024-09-18T14:50:17.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:18.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:18.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:18.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:50:18.226 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:50:18.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:18.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:50:18.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:19.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:19.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:19.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:50:19.536 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:50:19.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:19.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:50:19.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:20.047 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:50:20.038+0000 7f2ca0238540 -1 osd.0 0 log_to_monitors true 2024-09-18T14:50:20.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:20.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:20.852 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:50:20.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:50:20.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:20.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:50:21.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:22.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:22.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:22.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:50:22.196 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:50:22.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:22.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:50:22.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:23.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:23.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:23.500 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:50:23.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:50:23.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:23.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:50:23.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:24.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:24.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:24.849 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:50:24.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:50:24.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:24.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:50:25.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:26.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:26.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:26.166 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:50:26.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:50:26.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:26.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:50:26.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:27.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:27.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:27.486 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T14:50:27.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:50:27.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:27.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:50:27.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:28.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:28.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:28.867 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T14:50:28.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:50:28.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:28.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:50:29.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:30.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:30.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:30.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:50:30.181 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T14:50:30.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:30.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:50:30.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:31.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:31.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:31.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:50:31.503 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T14:50:31.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:31.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:50:31.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:32.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:32.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:32.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:50:32.888 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T14:50:32.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:32.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:50:33.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:34.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:34.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:34.327 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T14:50:34.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:50:34.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:34.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:50:34.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:35.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:35.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:35.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:50:35.839 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T14:50:35.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:35.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:50:36.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:37.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:37.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:37.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:50:37.269 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T14:50:37.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:37.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:50:37.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:38.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:38.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:38.666 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T14:50:38.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:50:38.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:38.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:50:39.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:40.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:40.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:40.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:50:40.361 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T14:50:40.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:40.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:50:41.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:42.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:42.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:42.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:50:42.341 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T14:50:42.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:42.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:50:42.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:43.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:43.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:43.883 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T14:50:43.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T14:50:43.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:43.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:50:44.197 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/138328,v1:127.0.0.1:6803/138328] [v2:127.0.0.1:6804/138328,v1:127.0.0.1:6805/138328] exists,up 9a72ec9c-4c15-4e49-8d80-69184823502e 2024-09-18T14:50:44.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:50:44.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:50:44.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:50:44.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:121: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-18T14:50:44.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T14:50:44.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:50:44.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T14:50:44.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:50:44.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T14:50:44.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T14:50:44.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:50:44.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:50:44.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:50:44.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T14:50:44.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T14:50:44.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:50:44.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:50:44.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T14:50:44.203 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:50:44.203 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:50:44.203 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:50:44.203 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:50:44.203 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:50:44.203 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:50:44.203 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:50:44.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:50:44.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:50:44.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:50:44.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:50:44.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T14:50:44.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T14:50:44.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:50:44.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:50:44.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:50:44.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:50:44.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:50:44.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-09-18T14:50:44.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T14:50:44.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:50:44.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ede9fb3b-74b6-41b6-a0da-f9d8925ddf7e 2024-09-18T14:50:44.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ede9fb3b-74b6-41b6-a0da-f9d8925ddf7e' 2024-09-18T14:50:44.205 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 ede9fb3b-74b6-41b6-a0da-f9d8925ddf7e 2024-09-18T14:50:44.206 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:50:44.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDE6Opm5BYlDRAAOuATGBRhtv6AokK2aB+OvA== 2024-09-18T14:50:44.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDE6Opm5BYlDRAAOuATGBRhtv6AokK2aB+OvA=="}' 2024-09-18T14:50:44.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ede9fb3b-74b6-41b6-a0da-f9d8925ddf7e -i td/osd-scrub-repair/1/new.json 2024-09-18T14:50:44.635 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:50:44.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-18T14:50:44.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQDE6Opm5BYlDRAAOuATGBRhtv6AokK2aB+OvA== --osd-uuid ede9fb3b-74b6-41b6-a0da-f9d8925ddf7e 2024-09-18T14:50:44.692 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:50:44.685+0000 7f45fccfa540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:50:44.710 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:50:44.701+0000 7f45fccfa540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:50:44.712 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:50:44.705+0000 7f45fccfa540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:50:44.713 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:50:44.705+0000 7f45fccfa540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T14:50:44.713 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:50:44.705+0000 7f45fccfa540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T14:50:44.713 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:50:44.705+0000 7f45fccfa540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T14:50:44.786 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:50:44.777+0000 7f45fccfa540 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-18T14:50:51.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-18T14:50:51.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:50:51.006 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T14:50:51.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T14:50:51.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:50:51.381 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T14:50:51.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T14:50:51.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:50:51.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-18T14:50:51.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:50:51.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:50:51.415 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:50:51.405+0000 7fa6cbb52540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:50:51.417 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:50:51.409+0000 7fa6cbb52540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:50:51.418 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:50:51.409+0000 7fa6cbb52540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:50:51.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T14:50:51.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:50:51.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:50:51.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:50:51.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:50:51.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:51.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:50:51.703 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:50:51.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:51.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:50:51.958 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:50:51.949+0000 7fa6cbb52540 -1 Falling back to public interface 2024-09-18T14:50:52.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:53.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:53.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:53.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:50:53.048 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:50:53.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:53.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:50:53.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:54.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:54.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:54.360 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:50:54.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:50:54.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:54.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:50:54.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:54.688 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:50:54.681+0000 7fa6cbb52540 -1 osd.1 0 log_to_monitors true 2024-09-18T14:50:55.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:55.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:55.674 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:50:55.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:50:55.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:55.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:50:56.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:57.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:57.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:57.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:50:57.069 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:50:57.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:57.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:50:57.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:58.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:58.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:58.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:50:58.381 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:50:58.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:58.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:50:58.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:59.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:59.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:59.751 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:50:59.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:50:59.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:59.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:51:00.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:51:01.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:51:01.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:51:01.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:51:01.067 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:51:01.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:51:01.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:51:01.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:51:02.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:51:02.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:51:02.382 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T14:51:02.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:51:02.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:51:02.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:51:02.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:51:03.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:51:03.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:51:03.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:51:03.765 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T14:51:03.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:51:03.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:51:04.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:51:05.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:51:05.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:51:05.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:51:05.080 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T14:51:05.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:51:05.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:51:05.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:51:06.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:51:06.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:51:06.541 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T14:51:06.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:51:06.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:51:06.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:51:07.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:51:08.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:51:08.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:51:08.082 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T14:51:08.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:51:08.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:51:08.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:51:08.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:51:09.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:51:09.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:51:09.483 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T14:51:09.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:51:09.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:51:09.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:51:09.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:51:10.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:51:10.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:51:10.810 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T14:51:10.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:51:10.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:51:10.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:51:11.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:51:12.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:51:12.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:51:12.261 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T14:51:12.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:51:12.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:51:12.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:51:12.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:51:13.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:51:13.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:51:13.686 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T14:51:13.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:51:13.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:51:13.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:51:14.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:51:15.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:51:15.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:51:15.478 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T14:51:15.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:51:15.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:51:15.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:51:15.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:51:16.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:51:16.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:51:16.836 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T14:51:16.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:51:16.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:51:16.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:51:17.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:51:18.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:51:18.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:51:18.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T14:51:18.442 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T14:51:18.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:51:18.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:51:18.770 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/140194,v1:127.0.0.1:6811/140194] [v2:127.0.0.1:6812/140194,v1:127.0.0.1:6813/140194] exists,up ede9fb3b-74b6-41b6-a0da-f9d8925ddf7e 2024-09-18T14:51:18.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:51:18.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:51:18.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:51:18.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:124: TEST_allow_repair_during_recovery: create_rbd_pool 2024-09-18T14:51:18.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T14:51:19.107 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' does not exist 2024-09-18T14:51:19.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T14:51:19.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T14:51:19.609 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' created 2024-09-18T14:51:19.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:51:20.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T14:51:22.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:125: TEST_allow_repair_during_recovery: wait_for_clean 2024-09-18T14:51:22.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:51:22.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:51:22.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:51:22.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T14:51:22.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:51:22.622 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:51:22.622 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:51:22.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:51:22.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:51:22.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:51:22.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:51:22.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:51:22.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:51:22.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:51:22.832 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:51:23.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:51:23.163 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T14:51:23.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:51:23.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:51:23.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:51:23.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836493 2024-09-18T14:51:23.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836493 2024-09-18T14:51:23.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836493' 2024-09-18T14:51:23.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:51:23.341 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:51:23.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672969 2024-09-18T14:51:23.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672969 2024-09-18T14:51:23.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836493 1-42949672969' 2024-09-18T14:51:23.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:51:23.487 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836493 2024-09-18T14:51:23.487 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:51:23.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:51:23.491 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836493 2024-09-18T14:51:23.491 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:51:23.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836493 2024-09-18T14:51:23.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-09-18T14:51:23.494 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836493 2024-09-18T14:51:23.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:51:23.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836492 -lt 21474836493 2024-09-18T14:51:23.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:51:24.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:51:24.811 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:51:25.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836494 -lt 21474836493 2024-09-18T14:51:25.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:51:25.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672969 2024-09-18T14:51:25.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:51:25.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:51:25.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672969 2024-09-18T14:51:25.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:51:25.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672969 2024-09-18T14:51:25.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-09-18T14:51:25.128 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672969 2024-09-18T14:51:25.129 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:51:25.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672969 -lt 42949672969 2024-09-18T14:51:25.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:51:25.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:51:25.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:51:25.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T14:51:25.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:51:25.899 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:51:25.899 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:51:25.899 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:51:25.899 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:51:25.899 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:51:25.899 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:51:26.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T14:51:26.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:51:26.242 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:51:26.242 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:51:26.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T14:51:26.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:51:26.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:51:26.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:127: TEST_allow_repair_during_recovery: add_something td/osd-scrub-repair rbd 2024-09-18T14:51:26.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T14:51:26.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=rbd 2024-09-18T14:51:26.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=SOMETHING 2024-09-18T14:51:26.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T14:51:26.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T14:51:26.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T14:51:27.036 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T14:51:27.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T14:51:27.522 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T14:51:27.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T14:51:27.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T14:51:27.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-18T14:51:27.862 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:128: TEST_allow_repair_during_recovery: get_not_primary rbd SOMETHING 2024-09-18T14:51:27.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-09-18T14:51:27.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-09-18T14:51:27.863 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-09-18T14:51:27.863 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-09-18T14:51:27.863 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T14:51:27.864 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-09-18T14:51:27.864 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T14:51:28.206 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T14:51:28.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-09-18T14:51:28.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T14:51:28.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:128: TEST_allow_repair_during_recovery: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-09-18T14:51:28.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:225: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-18T14:51:28.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local poolname=rbd 2024-09-18T14:51:28.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local osd=0 2024-09-18T14:51:28.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-18T14:51:28.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T14:51:28.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T14:51:28.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T14:51:28.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T14:51:28.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-18T14:51:28.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T14:51:28.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T14:51:28.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T14:51:28.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T14:51:28.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T14:51:28.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:51:28.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:51:28.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:51:28.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:51:28.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:51:28.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:51:28.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-18T14:51:28.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T14:51:28.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T14:51:28.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T14:51:28.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T14:51:28.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T14:51:28.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-18T14:51:30.499 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-18T14:51:31.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-18T14:51:31.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T14:51:31.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:51:31.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T14:51:31.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:51:31.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T14:51:31.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T14:51:31.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:51:31.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:51:31.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:51:31.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T14:51:31.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T14:51:31.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:51:31.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:51:31.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T14:51:31.036 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:51:31.036 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:51:31.037 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:51:31.038 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:51:31.038 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:51:31.038 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:51:31.038 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:51:31.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:51:31.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:51:31.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T14:51:31.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T14:51:31.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:51:31.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:51:31.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:51:31.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:51:31.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:51:31.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:51:31.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T14:51:31.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T14:51:31.040 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T14:51:31.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:51:31.041 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T14:51:31.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T14:51:31.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:51:31.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:51:31.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:51:31.071 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:51:31.061+0000 7f6c4c37a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:51:31.075 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:51:31.065+0000 7f6c4c37a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:51:31.077 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:51:31.069+0000 7f6c4c37a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:51:31.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T14:51:31.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:51:31.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:51:31.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:51:31.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:51:31.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:51:31.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:51:31.354 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T14:51:31.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:51:31.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:51:31.610 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:51:31.601+0000 7f6c4c37a540 -1 Falling back to public interface 2024-09-18T14:51:31.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:51:32.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:51:32.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:51:32.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:51:32.705 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:51:32.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:51:32.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:51:33.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:51:34.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:51:34.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:51:34.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:51:34.026 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T14:51:34.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:51:34.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:51:34.319 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:51:34.309+0000 7f6c4c37a540 -1 osd.0 21 log_to_monitors true 2024-09-18T14:51:34.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:51:35.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:51:35.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:51:35.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:51:35.470 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T14:51:35.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:51:35.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:51:35.786 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/142077,v1:127.0.0.1:6803/142077] [v2:127.0.0.1:6804/142077,v1:127.0.0.1:6805/142077] exists,up 9a72ec9c-4c15-4e49-8d80-69184823502e 2024-09-18T14:51:35.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:51:35.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:51:35.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:51:35.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T14:51:35.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:51:35.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:51:35.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:51:35.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T14:51:35.787 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:51:35.787 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:51:35.788 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:51:35.788 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:51:35.788 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:51:35.788 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:51:35.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:51:35.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:51:35.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:51:35.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:51:35.999 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:51:36.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:51:36.408 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T14:51:36.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:51:36.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:51:36.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:51:36.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=111669149698 2024-09-18T14:51:36.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 111669149698 2024-09-18T14:51:36.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-111669149698' 2024-09-18T14:51:36.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:51:36.592 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:51:36.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672972 2024-09-18T14:51:36.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672972 2024-09-18T14:51:36.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-111669149698 1-42949672972' 2024-09-18T14:51:36.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:51:36.735 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-111669149698 2024-09-18T14:51:36.735 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:51:36.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:51:36.738 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-111669149698 2024-09-18T14:51:36.738 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:51:36.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=111669149698 2024-09-18T14:51:36.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-09-18T14:51:36.740 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 111669149698 2024-09-18T14:51:36.740 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:51:37.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 111669149698 -lt 111669149698 2024-09-18T14:51:37.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:51:37.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672972 2024-09-18T14:51:37.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:51:37.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:51:37.065 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672972 2024-09-18T14:51:37.065 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:51:37.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672972 2024-09-18T14:51:37.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-09-18T14:51:37.066 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 42949672972 2024-09-18T14:51:37.067 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:51:37.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672972 -lt 42949672972 2024-09-18T14:51:37.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:51:37.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:51:37.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:51:37.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T14:51:37.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:51:37.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:51:37.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:51:37.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:51:37.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:51:37.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:51:37.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:51:38.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T14:51:38.163 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:51:38.163 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:51:38.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:51:38.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T14:51:38.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:51:38.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:51:38.526 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-09-18T14:51:38.526 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-09-18T14:51:38.526 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T14:51:38.526 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-09-18T14:51:38.527 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T14:51:38.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: local pg=1.3 2024-09-18T14:51:38.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: repair 1.3 2024-09-18T14:51:38.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=1.3 2024-09-18T14:51:38.895 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 1.3 2024-09-18T14:51:38.896 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T14:51:38.896 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:51:38.896 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:51:38.896 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T14:51:39.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T14:51:19.505354+0000 2024-09-18T14:51:39.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 1.3 2024-09-18T14:51:39.530 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 1.3 on osd.1 to repair 2024-09-18T14:51:39.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 1.3 2024-09-18T14:51:19.505354+0000 2024-09-18T14:51:39.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.3 2024-09-18T14:51:39.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T14:51:19.505354+0000 2024-09-18T14:51:39.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T14:51:39.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T14:51:39.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T14:51:39.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-18T14:51:39.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T14:51:39.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:51:39.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:51:39.540 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T14:51:39.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:51:19.505354+0000 '>' 2024-09-18T14:51:19.505354+0000 2024-09-18T14:51:39.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T14:51:40.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T14:51:40.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T14:51:40.877 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-18T14:51:40.877 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T14:51:40.877 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:51:40.877 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:51:40.878 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T14:51:41.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:51:19.505354+0000 '>' 2024-09-18T14:51:19.505354+0000 2024-09-18T14:51:41.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T14:51:42.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T14:51:42.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T14:51:42.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-18T14:51:42.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T14:51:42.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:51:42.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:51:42.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T14:51:42.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:51:19.505354+0000 '>' 2024-09-18T14:51:19.505354+0000 2024-09-18T14:51:42.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T14:51:43.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T14:51:43.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T14:51:43.510 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-18T14:51:43.510 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T14:51:43.510 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:51:43.511 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:51:43.511 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T14:51:43.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:51:19.505354+0000 '>' 2024-09-18T14:51:19.505354+0000 2024-09-18T14:51:43.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T14:51:44.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T14:51:44.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T14:51:44.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-18T14:51:44.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T14:51:44.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:51:44.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:51:44.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T14:51:45.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:51:19.505354+0000 '>' 2024-09-18T14:51:19.505354+0000 2024-09-18T14:51:45.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T14:51:46.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T14:51:46.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T14:51:46.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-18T14:51:46.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T14:51:46.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:51:46.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:51:46.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T14:51:46.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:51:19.505354+0000 '>' 2024-09-18T14:51:19.505354+0000 2024-09-18T14:51:46.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T14:51:47.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T14:51:47.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T14:51:47.473 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-18T14:51:47.473 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T14:51:47.473 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:51:47.473 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:51:47.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T14:51:47.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:51:19.505354+0000 '>' 2024-09-18T14:51:19.505354+0000 2024-09-18T14:51:47.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T14:51:48.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T14:51:48.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T14:51:48.804 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-18T14:51:48.804 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T14:51:48.804 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:51:48.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:51:48.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T14:51:49.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:51:43.529863+0000 '>' 2024-09-18T14:51:19.505354+0000 2024-09-18T14:51:49.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T14:51:49.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:241: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T14:51:49.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T14:51:49.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T14:51:49.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T14:51:49.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T14:51:49.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T14:51:49.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T14:51:49.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T14:51:49.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T14:51:49.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T14:51:49.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T14:51:49.116 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:51:49.116 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:51:49.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:51:49.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:51:49.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:51:49.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:51:49.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T14:51:49.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T14:51:49.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T14:51:49.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T14:51:49.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T14:51:49.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T14:51:49.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-18T14:51:50.949 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-18T14:51:50.949 INFO:tasks.workunit.client.0.smithi046.stdout:_ 2024-09-18T14:51:50.949 INFO:tasks.workunit.client.0.smithi046.stdout:snapset 2024-09-18T14:51:51.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T14:51:51.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T14:51:51.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:51:51.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T14:51:51.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:51:51.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T14:51:51.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T14:51:51.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:51:51.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:51:51.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:51:51.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T14:51:51.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T14:51:51.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:51:51.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:51:51.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T14:51:51.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:51:51.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:51:51.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:51:51.243 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:51:51.243 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:51:51.243 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:51:51.243 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:51:51.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:51:51.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:51:51.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T14:51:51.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T14:51:51.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:51:51.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:51:51.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:51:51.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:51:51.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:51:51.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:51:51.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T14:51:51.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T14:51:51.244 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T14:51:51.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:51:51.245 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T14:51:51.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T14:51:51.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:51:51.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:51:51.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:51:51.290 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:51:51.280+0000 7faa62851540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:51:51.296 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:51:51.284+0000 7faa62851540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:51:51.297 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:51:51.288+0000 7faa62851540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:51:51.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T14:51:51.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:51:51.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:51:51.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:51:51.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:51:51.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:51:51.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:51:51.556 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T14:51:51.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:51:51.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:51:51.606 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:51:51.596+0000 7faa62851540 -1 Falling back to public interface 2024-09-18T14:51:51.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:51:52.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:51:52.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:51:52.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:51:52.887 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:51:52.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:51:52.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:51:53.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:51:54.194 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:51:54.184+0000 7faa62851540 -1 osd.1 28 log_to_monitors true 2024-09-18T14:51:54.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:51:54.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:51:54.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:51:54.199 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T14:51:54.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:51:54.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:51:54.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:51:55.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:51:55.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:51:55.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:51:55.595 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T14:51:55.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:51:55.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:51:55.953 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 32 up_thru 32 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/143635,v1:127.0.0.1:6811/143635] [v2:127.0.0.1:6812/143635,v1:127.0.0.1:6813/143635] exists,up ede9fb3b-74b6-41b6-a0da-f9d8925ddf7e 2024-09-18T14:51:55.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:51:55.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:51:55.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:51:55.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T14:51:55.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:51:55.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:51:55.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:51:55.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T14:51:55.954 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:51:55.954 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:51:55.954 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:51:55.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:51:55.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:51:55.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:51:56.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:51:56.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:51:56.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:51:56.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:51:56.173 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:51:56.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:51:56.493 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T14:51:56.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:51:56.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:51:56.494 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:51:56.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=111669149703 2024-09-18T14:51:56.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 111669149703 2024-09-18T14:51:56.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-111669149703' 2024-09-18T14:51:56.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:51:56.683 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:51:56.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953474 2024-09-18T14:51:56.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953474 2024-09-18T14:51:56.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-111669149703 1-137438953474' 2024-09-18T14:51:56.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:51:56.825 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-111669149703 2024-09-18T14:51:56.826 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:51:56.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:51:56.829 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-111669149703 2024-09-18T14:51:56.829 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:51:56.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=111669149703 2024-09-18T14:51:56.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 111669149703' 2024-09-18T14:51:56.831 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 111669149703 2024-09-18T14:51:56.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:51:57.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 111669149703 -lt 111669149703 2024-09-18T14:51:57.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:51:57.169 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-137438953474 2024-09-18T14:51:57.170 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:51:57.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:51:57.173 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-137438953474 2024-09-18T14:51:57.173 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:51:57.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953474 2024-09-18T14:51:57.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-09-18T14:51:57.175 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 137438953474 2024-09-18T14:51:57.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:51:57.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 137438953474 2024-09-18T14:51:57.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:51:58.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:51:58.516 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:51:58.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 137438953474 2024-09-18T14:51:58.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:51:59.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-18T14:51:59.825 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:52:00.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953474 -lt 137438953474 2024-09-18T14:52:00.140 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:52:00.141 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:52:00.141 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:52:00.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T14:52:00.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:52:00.502 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:52:00.502 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:52:00.502 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:52:00.503 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:52:00.503 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:52:00.503 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:52:00.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T14:52:00.843 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:52:00.843 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:52:00.844 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:52:01.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T14:52:01.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:52:01.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:52:01.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-09-18T14:52:01.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-18T14:52:01.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T14:52:01.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T14:52:01.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:52:01.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T14:52:01.287 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:52:01.287 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:52:01.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:52:01.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:52:01.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:52:01.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:52:01.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:52:01.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:52:01.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:52:01.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:52:01.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:52:01.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:52:01.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:52:01.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:52:01.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:52:01.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:52:01.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:52:01.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:52:01.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T14:52:01.431 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:52:01.431 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:52:01.431 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:52:01.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T14:52:01.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:52:01.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:52:01.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T14:52:01.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T14:52:01.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T14:52:01.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T14:52:01.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T14:52:01.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:52:01.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T14:52:01.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:52:01.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:52:01.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:52:01.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:52:01.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:52:01.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:52:01.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:52:01.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:52:01.442 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:52:01.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:52:01.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:52:01.444 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:52:01.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:52:01.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:52:01.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:52:01.447 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:52:01.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:52:01.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:52:01.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T14:52:01.452 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:52:01.452 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:52:01.452 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:52:01.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T14:52:01.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:52:01.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:52:01.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T14:52:01.457 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:52:01.457 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:52:01.457 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:52:01.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T14:52:01.460 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:52:01.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T14:52:01.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:52:01.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T14:52:01.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_auto_repair_bluestore_basic td/osd-scrub-repair 2024-09-18T14:52:01.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:442: TEST_auto_repair_bluestore_basic: local dir=td/osd-scrub-repair 2024-09-18T14:52:01.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:443: TEST_auto_repair_bluestore_basic: local poolname=testpool 2024-09-18T14:52:01.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:446: TEST_auto_repair_bluestore_basic: run_mon td/osd-scrub-repair a 2024-09-18T14:52:01.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T14:52:01.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:52:01.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:52:01.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:52:01.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T14:52:01.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-18T14:52:01.873 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:52:01.873 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:52:01.873 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:52:01.873 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:52:01.874 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:52:01.874 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:52:01.874 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:52:01.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T14:52:02.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:52:02.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:52:02.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:52:02.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:52:02.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:52:02.149 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:52:02.149 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:52:02.149 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:52:02.149 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:52:02.150 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:52:02.150 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:52:02.150 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:52:02.151 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T14:52:02.152 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:52:02.152 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T14:52:02.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:52:02.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:52:02.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:52:02.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:52:02.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:52:02.254 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:52:02.254 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:52:02.255 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:52:02.255 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:52:02.255 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:52:02.255 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:52:02.255 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T14:52:02.255 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:52:02.255 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T14:52:02.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_basic: run_mgr td/osd-scrub-repair x 2024-09-18T14:52:02.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T14:52:02.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:52:02.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:52:02.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:52:02.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T14:52:02.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:52:03.206 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:52:03.206 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:52:03.206 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:52:03.207 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:52:03.207 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:52:03.207 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:52:03.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:52:03.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:52:03.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:52:03.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:450: TEST_auto_repair_bluestore_basic: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T14:52:03.245 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:451: TEST_auto_repair_bluestore_basic: seq 0 2 2024-09-18T14:52:03.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:451: TEST_auto_repair_bluestore_basic: for id in $(seq 0 2) 2024-09-18T14:52:03.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:452: TEST_auto_repair_bluestore_basic: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:52:03.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T14:52:03.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:52:03.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:52:03.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:52:03.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T14:52:03.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T14:52:03.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:52:03.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:52:03.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:52:03.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T14:52:03.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T14:52:03.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:52:03.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:52:03.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T14:52:03.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:52:03.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:52:03.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:52:03.252 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:52:03.252 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:52:03.252 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:52:03.252 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:52:03.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:52:03.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:52:03.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:52:03.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:52:03.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T14:52:03.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T14:52:03.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:52:03.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:52:03.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:52:03.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:52:03.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:52:03.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T14:52:03.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T14:52:03.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:52:03.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=49025be4-3064-415c-9430-495d941953eb 2024-09-18T14:52:03.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 49025be4-3064-415c-9430-495d941953eb' 2024-09-18T14:52:03.255 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 49025be4-3064-415c-9430-495d941953eb 2024-09-18T14:52:03.255 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:52:03.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAT6epmPXoCEBAALoGMdYm7o1WqRoNHlk7d3Q== 2024-09-18T14:52:03.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAT6epmPXoCEBAALoGMdYm7o1WqRoNHlk7d3Q=="}' 2024-09-18T14:52:03.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 49025be4-3064-415c-9430-495d941953eb -i td/osd-scrub-repair/0/new.json 2024-09-18T14:52:03.580 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:52:03.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-18T14:52:03.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAT6epmPXoCEBAALoGMdYm7o1WqRoNHlk7d3Q== --osd-uuid 49025be4-3064-415c-9430-495d941953eb 2024-09-18T14:52:03.627 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:52:03.616+0000 7f21c0f16540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:52:03.675 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:52:03.664+0000 7f21c0f16540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:52:03.677 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:52:03.668+0000 7f21c0f16540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:52:03.679 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:52:03.668+0000 7f21c0f16540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T14:52:03.679 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:52:03.668+0000 7f21c0f16540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T14:52:03.679 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:52:03.668+0000 7f21c0f16540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T14:52:03.723 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:52:03.712+0000 7f21c0f16540 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-18T14:52:09.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T14:52:09.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:52:09.959 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T14:52:09.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:52:09.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:52:10.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:52:10.342 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T14:52:10.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:52:10.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:52:10.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:52:10.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:52:10.373 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:52:10.364+0000 7f00a25d6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:52:10.376 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:52:10.368+0000 7f00a25d6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:52:10.379 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:52:10.368+0000 7f00a25d6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:52:10.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:52:10.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:52:10.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:52:10.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:52:10.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:52:10.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:10.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:52:10.641 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:52:10.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:10.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:52:10.690 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:52:10.680+0000 7f00a25d6540 -1 Falling back to public interface 2024-09-18T14:52:10.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:11.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:11.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:11.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:52:11.937 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:52:11.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:11.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:52:12.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:13.206 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:52:13.196+0000 7f00a25d6540 -1 osd.0 0 log_to_monitors true 2024-09-18T14:52:13.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:13.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:13.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:52:13.266 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:52:13.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:13.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:52:13.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:14.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:14.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:14.601 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:52:14.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:52:14.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:14.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:52:14.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:15.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:15.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:15.956 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:52:15.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:52:15.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:15.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:52:16.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:17.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:17.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:17.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:52:17.271 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:52:17.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:17.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:52:17.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:18.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:18.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:18.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:52:18.587 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:52:18.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:18.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:52:18.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:19.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:19.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:19.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:52:19.911 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:52:19.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:19.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:52:20.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:21.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:21.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:21.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:52:21.227 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T14:52:21.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:21.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:52:21.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:22.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:22.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:22.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:52:22.543 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T14:52:22.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:22.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:52:22.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:23.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:23.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:23.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:52:23.959 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T14:52:23.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:23.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:52:24.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:25.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:25.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:25.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:52:25.268 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T14:52:25.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:25.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:52:25.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:26.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:26.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:26.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:52:26.593 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T14:52:26.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:26.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:52:27.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:28.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:28.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:28.216 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T14:52:28.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:52:28.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:28.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:52:28.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:29.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:29.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:29.644 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T14:52:29.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:52:29.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:29.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:52:30.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:31.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:31.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:31.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:52:31.319 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T14:52:31.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:31.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:52:32.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:33.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:33.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:33.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:52:33.059 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T14:52:33.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:33.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:52:33.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:34.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:34.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:34.536 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T14:52:34.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:52:34.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:34.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:52:35.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:36.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:36.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:36.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:52:36.032 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T14:52:36.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:36.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:52:36.383 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/145714,v1:127.0.0.1:6803/145714] [v2:127.0.0.1:6804/145714,v1:127.0.0.1:6805/145714] exists,up 49025be4-3064-415c-9430-495d941953eb 2024-09-18T14:52:36.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:52:36.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:52:36.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:52:36.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:451: TEST_auto_repair_bluestore_basic: for id in $(seq 0 2) 2024-09-18T14:52:36.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:452: TEST_auto_repair_bluestore_basic: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:52:36.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T14:52:36.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:52:36.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T14:52:36.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:52:36.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T14:52:36.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T14:52:36.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:52:36.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:52:36.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:52:36.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T14:52:36.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T14:52:36.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:52:36.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:52:36.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T14:52:36.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:52:36.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:52:36.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:52:36.389 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:52:36.389 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:52:36.389 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:52:36.389 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:52:36.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:52:36.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:52:36.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:52:36.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:52:36.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T14:52:36.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T14:52:36.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:52:36.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:52:36.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:52:36.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:52:36.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:52:36.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T14:52:36.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T14:52:36.391 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:52:36.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=14119483-7ebe-4afe-acc2-57c643a5309d 2024-09-18T14:52:36.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 14119483-7ebe-4afe-acc2-57c643a5309d' 2024-09-18T14:52:36.392 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 14119483-7ebe-4afe-acc2-57c643a5309d 2024-09-18T14:52:36.392 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:52:36.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA06epmDrZIGBAA3HC3vJSnCn7mNlAZZrmd+w== 2024-09-18T14:52:36.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA06epmDrZIGBAA3HC3vJSnCn7mNlAZZrmd+w=="}' 2024-09-18T14:52:36.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 14119483-7ebe-4afe-acc2-57c643a5309d -i td/osd-scrub-repair/1/new.json 2024-09-18T14:52:36.880 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:52:36.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-18T14:52:36.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQA06epmDrZIGBAA3HC3vJSnCn7mNlAZZrmd+w== --osd-uuid 14119483-7ebe-4afe-acc2-57c643a5309d 2024-09-18T14:52:36.938 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:52:36.928+0000 7fe9cf927540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:52:36.942 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:52:36.932+0000 7fe9cf927540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:52:36.944 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:52:36.936+0000 7fe9cf927540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:52:36.945 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:52:36.936+0000 7fe9cf927540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T14:52:36.945 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:52:36.936+0000 7fe9cf927540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T14:52:36.945 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:52:36.936+0000 7fe9cf927540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T14:52:37.006 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:52:36.996+0000 7fe9cf927540 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-18T14:52:43.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-18T14:52:43.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:52:43.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T14:52:43.292 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T14:52:43.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:52:43.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T14:52:43.684 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T14:52:43.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:52:43.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:52:43.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:52:43.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:52:43.717 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:52:43.708+0000 7f8814190540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:52:43.720 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:52:43.712+0000 7f8814190540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:52:43.723 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:52:43.712+0000 7f8814190540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:52:44.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T14:52:44.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:52:44.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:52:44.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:52:44.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:52:44.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:44.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:52:44.031 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:52:44.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:44.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:52:44.285 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:52:44.276+0000 7f8814190540 -1 Falling back to public interface 2024-09-18T14:52:44.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:45.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:45.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:45.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:52:45.414 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:52:45.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:45.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:52:45.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:46.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:46.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:46.738 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:52:46.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:52:46.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:46.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:52:47.038 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:52:47.028+0000 7f8814190540 -1 osd.1 0 log_to_monitors true 2024-09-18T14:52:47.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:48.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:48.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:48.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:52:48.062 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:52:48.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:48.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:52:48.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:49.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:49.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:49.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:52:49.372 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:52:49.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:52:49.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:49.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:50.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:50.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:50.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:52:50.930 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:52:50.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:50.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:52:51.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:52.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:52.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:52.239 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:52:52.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:52:52.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:52.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:52:52.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:53.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:53.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:53.545 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:52:53.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:52:53.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:53.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:52:53.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:54.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:54.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:54.951 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T14:52:54.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:52:54.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:54.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:52:55.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:56.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:56.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:56.266 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T14:52:56.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:52:56.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:56.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:52:56.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:57.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:57.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:57.587 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T14:52:57.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:52:57.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:57.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:52:57.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:52:58.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:52:58.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:52:58.997 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T14:52:58.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:52:58.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:52:58.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:52:59.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:00.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:00.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:00.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:53:00.308 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T14:53:00.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:00.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:53:00.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:01.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:01.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:01.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:53:01.677 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T14:53:01.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:01.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:53:02.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:03.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:03.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:03.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:53:03.066 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T14:53:03.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:03.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:53:03.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:04.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:04.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:04.682 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T14:53:04.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:53:04.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:04.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:53:05.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:06.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:06.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:06.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:53:06.276 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T14:53:06.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:06.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:53:06.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:07.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:07.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:07.749 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T14:53:07.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:53:07.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:07.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:53:08.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:09.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:09.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:09.462 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T14:53:09.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:53:09.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:09.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:53:09.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:10.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:10.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:10.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T14:53:10.942 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T14:53:10.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:10.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:53:11.267 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/147493,v1:127.0.0.1:6811/147493] [v2:127.0.0.1:6812/147493,v1:127.0.0.1:6813/147493] exists,up 14119483-7ebe-4afe-acc2-57c643a5309d 2024-09-18T14:53:11.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:53:11.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:53:11.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:53:11.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:451: TEST_auto_repair_bluestore_basic: for id in $(seq 0 2) 2024-09-18T14:53:11.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:452: TEST_auto_repair_bluestore_basic: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:53:11.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T14:53:11.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:53:11.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T14:53:11.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:53:11.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-18T14:53:11.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T14:53:11.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:53:11.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:53:11.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:53:11.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T14:53:11.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T14:53:11.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:53:11.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:53:11.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T14:53:11.270 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:53:11.270 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:53:11.270 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:53:11.272 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:53:11.272 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:53:11.272 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:53:11.272 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:53:11.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:53:11.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:53:11.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:53:11.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:53:11.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T14:53:11.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T14:53:11.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:53:11.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:53:11.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:53:11.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:53:11.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:53:11.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T14:53:11.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-18T14:53:11.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:53:11.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=66c6e723-08c7-4e85-b2e0-04f558956f3f 2024-09-18T14:53:11.275 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 66c6e723-08c7-4e85-b2e0-04f558956f3f 2024-09-18T14:53:11.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 66c6e723-08c7-4e85-b2e0-04f558956f3f' 2024-09-18T14:53:11.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:53:11.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBX6epmoMRgERAAqDm4yQKY8OevTJcbcOB7Vw== 2024-09-18T14:53:11.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBX6epmoMRgERAAqDm4yQKY8OevTJcbcOB7Vw=="}' 2024-09-18T14:53:11.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 66c6e723-08c7-4e85-b2e0-04f558956f3f -i td/osd-scrub-repair/2/new.json 2024-09-18T14:53:11.737 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:53:11.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-18T14:53:11.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBX6epmoMRgERAAqDm4yQKY8OevTJcbcOB7Vw== --osd-uuid 66c6e723-08c7-4e85-b2e0-04f558956f3f 2024-09-18T14:53:11.794 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:53:11.784+0000 7fcd4dde1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:53:11.797 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:53:11.788+0000 7fcd4dde1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:53:11.798 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:53:11.788+0000 7fcd4dde1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:53:11.799 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:53:11.788+0000 7fcd4dde1540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T14:53:11.799 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:53:11.788+0000 7fcd4dde1540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T14:53:11.799 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:53:11.788+0000 7fcd4dde1540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T14:53:11.862 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:53:11.856+0000 7fcd4dde1540 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-18T14:53:18.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-18T14:53:18.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:53:18.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T14:53:18.471 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T14:53:18.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:53:18.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T14:53:18.869 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T14:53:18.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:53:18.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:53:18.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:53:18.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:53:18.900 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:53:18.896+0000 7f4f4804c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:53:18.903 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:53:18.896+0000 7f4f4804c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:53:18.904 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:53:18.900+0000 7f4f4804c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:53:19.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T14:53:19.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:53:19.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:53:19.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:53:19.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:53:19.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:19.202 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:53:19.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:53:19.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:19.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:53:19.438 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:53:19.432+0000 7f4f4804c540 -1 Falling back to public interface 2024-09-18T14:53:19.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:20.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:20.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:20.557 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:53:20.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:53:20.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:20.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:53:20.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:21.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:21.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:21.873 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:53:21.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:53:21.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:21.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:53:22.154 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:53:22.148+0000 7f4f4804c540 -1 osd.2 0 log_to_monitors true 2024-09-18T14:53:22.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:23.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:23.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:23.196 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:53:23.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:53:23.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:23.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:53:23.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:24.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:24.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:24.603 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:53:24.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:53:24.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:24.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:53:25.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:26.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:26.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:26.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:53:26.071 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:53:26.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:53:26.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:26.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:27.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:27.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:27.449 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:53:27.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:53:27.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:27.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:53:27.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:28.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:28.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:28.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:53:28.949 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:53:28.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:28.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:53:29.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:30.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:30.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:30.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:53:30.272 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T14:53:30.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:30.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:53:30.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:31.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:31.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:31.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:53:31.595 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T14:53:31.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:31.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:53:31.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:32.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:32.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:32.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:53:32.990 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T14:53:32.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:32.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:53:33.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:34.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:34.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:34.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:53:34.308 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T14:53:34.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:34.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:53:34.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:35.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:35.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:35.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:53:35.619 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T14:53:35.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:35.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:53:36.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:37.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:37.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:37.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:53:37.317 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T14:53:37.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:37.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:53:37.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:38.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:39.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:39.000 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T14:53:39.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:53:39.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:39.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:53:39.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:40.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:40.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:40.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:53:40.452 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T14:53:40.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:40.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:53:40.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:41.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:41.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:41.883 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T14:53:41.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:53:41.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:41.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:53:42.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:43.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:43.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:43.294 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T14:53:43.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:53:43.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:43.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:53:44.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:45.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:45.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:45.016 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T14:53:45.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:53:45.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:45.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:53:45.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:46.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:46.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:46.451 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T14:53:46.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T14:53:46.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:46.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:53:46.798 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/149297,v1:127.0.0.1:6819/149297] [v2:127.0.0.1:6820/149297,v1:127.0.0.1:6821/149297] exists,up 66c6e723-08c7-4e85-b2e0-04f558956f3f 2024-09-18T14:53:46.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:53:46.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:53:46.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:53:46.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:455: TEST_auto_repair_bluestore_basic: create_pool testpool 1 1 2024-09-18T14:53:46.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-09-18T14:53:47.313 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'testpool' created 2024-09-18T14:53:47.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:53:48.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_basic: ceph osd pool set testpool size 2 2024-09-18T14:53:49.066 INFO:tasks.workunit.client.0.smithi046.stderr:set pool 1 size to 2 2024-09-18T14:53:49.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:457: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-09-18T14:53:49.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:53:49.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:53:49.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:53:49.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T14:53:49.080 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:53:49.081 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:53:49.081 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:53:49.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:53:49.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:53:49.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:53:49.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:53:49.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:53:49.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:53:49.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:53:49.297 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:53:49.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:53:49.847 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:53:49.847 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T14:53:49.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:53:49.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:53:49.848 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:53:49.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836498 2024-09-18T14:53:49.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836498 2024-09-18T14:53:49.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498' 2024-09-18T14:53:49.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:53:49.991 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:53:50.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672972 2024-09-18T14:53:50.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672972 2024-09-18T14:53:50.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498 1-42949672972' 2024-09-18T14:53:50.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:53:50.116 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:53:50.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509446 2024-09-18T14:53:50.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509446 2024-09-18T14:53:50.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498 1-42949672972 2-64424509446' 2024-09-18T14:53:50.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:53:50.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836498 2024-09-18T14:53:50.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:53:50.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:53:50.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836498 2024-09-18T14:53:50.252 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:53:50.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836498 2024-09-18T14:53:50.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-09-18T14:53:50.254 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836498 2024-09-18T14:53:50.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:53:50.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836497 -lt 21474836498 2024-09-18T14:53:50.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:53:51.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:53:51.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:53:51.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836498 -lt 21474836498 2024-09-18T14:53:51.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:53:51.900 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672972 2024-09-18T14:53:51.900 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:53:51.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:53:51.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672972 2024-09-18T14:53:51.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:53:51.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672972 2024-09-18T14:53:51.905 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672972 2024-09-18T14:53:51.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-09-18T14:53:51.906 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:53:52.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672972 -lt 42949672972 2024-09-18T14:53:52.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:53:52.265 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509446 2024-09-18T14:53:52.265 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:53:52.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:53:52.268 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509446 2024-09-18T14:53:52.268 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:53:52.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509446 2024-09-18T14:53:52.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-09-18T14:53:52.271 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509446 2024-09-18T14:53:52.271 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:53:52.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509446 -lt 64424509446 2024-09-18T14:53:52.602 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:53:52.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:52.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:52.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T14:53:52.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:53:52.975 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:53:52.975 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:53:52.976 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:53:52.976 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:53:52.976 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:53:52.976 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:53:53.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T14:53:53.271 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:53:53.272 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:53.272 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:53.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T14:53:53.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:53:53.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:53:53.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: local payload=ABCDEF 2024-09-18T14:53:53.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:461: TEST_auto_repair_bluestore_basic: echo ABCDEF 2024-09-18T14:53:53.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:462: TEST_auto_repair_bluestore_basic: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-18T14:53:53.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:466: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-09-18T14:53:53.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-09-18T14:53:53.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-09-18T14:53:53.858 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-09-18T14:53:53.858 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-09-18T14:53:53.858 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T14:53:53.858 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T14:53:53.858 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T14:53:54.170 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T14:53:54.170 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T14:53:54.170 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T14:53:54.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:466: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-18T14:53:54.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T14:53:54.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T14:53:54.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T14:53:54.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T14:53:54.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-18T14:53:54.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T14:53:54.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T14:53:54.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T14:53:54.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T14:53:54.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T14:53:54.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:53:54.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:53:54.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:53:54.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:53:54.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:53:54.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:53:54.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-18T14:53:54.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T14:53:54.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T14:53:54.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T14:53:54.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T14:53:54.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T14:53:54.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-18T14:53:56.582 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-18T14:53:57.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:53:57.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T14:53:57.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:53:57.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T14:53:57.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:53:57.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T14:53:57.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T14:53:57.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:53:57.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:53:57.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:53:57.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T14:53:57.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T14:53:57.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:53:57.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:53:57.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T14:53:57.120 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:53:57.120 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:53:57.120 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:53:57.123 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:53:57.123 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:53:57.123 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:53:57.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:53:57.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:53:57.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:53:57.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T14:53:57.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T14:53:57.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:53:57.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:53:57.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:53:57.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:53:57.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:53:57.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T14:53:57.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T14:53:57.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T14:53:57.124 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T14:53:57.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:53:57.124 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T14:53:57.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T14:53:57.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:53:57.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:53:57.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:53:57.158 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:53:57.151+0000 7f3ba0f62540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:53:57.161 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:53:57.155+0000 7f3ba0f62540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:53:57.163 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:53:57.155+0000 7f3ba0f62540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:53:57.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T14:53:57.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:53:57.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:53:57.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:53:57.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:53:57.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:57.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:53:57.436 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T14:53:57.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:57.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:53:57.697 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:53:57.691+0000 7f3ba0f62540 -1 Falling back to public interface 2024-09-18T14:53:57.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:53:58.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:53:58.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:53:58.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:53:58.739 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:53:58.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:53:58.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:53:59.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:54:00.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:54:00.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:54:00.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:54:00.057 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T14:54:00.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:54:00.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:54:00.362 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:54:00.355+0000 7f3ba0f62540 -1 osd.0 21 log_to_monitors true 2024-09-18T14:54:00.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:54:01.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:54:01.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:54:01.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:54:01.385 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T14:54:01.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:54:01.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:54:01.695 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/151171,v1:127.0.0.1:6803/151171] [v2:127.0.0.1:6804/151171,v1:127.0.0.1:6805/151171] exists,up 49025be4-3064-415c-9430-495d941953eb 2024-09-18T14:54:01.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:54:01.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:54:01.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:54:01.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T14:54:01.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:54:01.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:54:01.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:54:01.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T14:54:01.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:54:01.697 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:54:01.697 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:54:01.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:54:01.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:54:01.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:54:01.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:54:01.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:54:01.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:54:01.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:54:01.914 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:54:02.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:54:02.239 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:54:02.239 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T14:54:02.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:54:02.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:54:02.239 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:54:02.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182402 2024-09-18T14:54:02.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182402 2024-09-18T14:54:02.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182402' 2024-09-18T14:54:02.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:54:02.378 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:54:02.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672976 2024-09-18T14:54:02.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672976 2024-09-18T14:54:02.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182402 1-42949672976' 2024-09-18T14:54:02.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:54:02.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:54:02.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509450 2024-09-18T14:54:02.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509450 2024-09-18T14:54:02.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182402 1-42949672976 2-64424509450' 2024-09-18T14:54:02.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:54:02.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-107374182402 2024-09-18T14:54:02.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:54:02.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:54:02.695 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-107374182402 2024-09-18T14:54:02.696 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:54:02.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182402 2024-09-18T14:54:02.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-09-18T14:54:02.698 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 107374182402 2024-09-18T14:54:02.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:54:03.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182402 -lt 107374182402 2024-09-18T14:54:03.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:54:03.013 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672976 2024-09-18T14:54:03.013 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:54:03.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:54:03.016 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672976 2024-09-18T14:54:03.016 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:54:03.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672976 2024-09-18T14:54:03.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-18T14:54:03.018 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 42949672976 2024-09-18T14:54:03.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:54:03.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672976 -lt 42949672976 2024-09-18T14:54:03.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:54:03.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509450 2024-09-18T14:54:03.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:54:03.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:54:03.364 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509450 2024-09-18T14:54:03.364 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:54:03.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509450 2024-09-18T14:54:03.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-09-18T14:54:03.366 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509450 2024-09-18T14:54:03.367 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:54:03.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509449 -lt 64424509450 2024-09-18T14:54:03.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:54:04.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:54:04.719 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:54:05.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509450 -lt 64424509450 2024-09-18T14:54:05.028 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:54:05.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:05.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:05.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T14:54:05.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:05.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:05.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:05.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:05.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:05.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:05.426 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:05.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T14:54:05.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:05.772 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:05.772 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:06.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T14:54:06.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:54:06.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:54:06.129 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:468: TEST_auto_repair_bluestore_basic: get_pg testpool SOMETHING 2024-09-18T14:54:06.129 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-09-18T14:54:06.129 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T14:54:06.130 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-09-18T14:54:06.130 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T14:54:06.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:468: TEST_auto_repair_bluestore_basic: local pgid=1.0 2024-09-18T14:54:06.479 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:469: TEST_auto_repair_bluestore_basic: get_primary testpool SOMETHING 2024-09-18T14:54:06.479 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-09-18T14:54:06.479 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T14:54:06.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T14:54:06.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T14:54:06.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:469: TEST_auto_repair_bluestore_basic: local primary=1 2024-09-18T14:54:06.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:470: TEST_auto_repair_bluestore_basic: get_last_scrub_stamp 1.0 2024-09-18T14:54:06.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T14:54:06.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:54:06.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:54:06.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T14:54:07.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:470: TEST_auto_repair_bluestore_basic: local last_scrub_stamp=2024-09-18T14:53:47.214999+0000 2024-09-18T14:54:07.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:471: TEST_auto_repair_bluestore_basic: ceph tell 1.0 deep_scrub 2024-09-18T14:54:07.254 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:54:07.254 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": true, 2024-09-18T14:54:07.254 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:52:27.250244+0000" 2024-09-18T14:54:07.254 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:54:07.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:472: TEST_auto_repair_bluestore_basic: ceph tell 1.0 scrub 2024-09-18T14:54:07.398 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:54:07.398 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:54:07.399 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:52:27.394548+0000" 2024-09-18T14:54:07.399 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:54:07.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:475: TEST_auto_repair_bluestore_basic: wait_for_scrub 1.0 2024-09-18T14:53:47.214999+0000 2024-09-18T14:54:07.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-18T14:54:07.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T14:53:47.214999+0000 2024-09-18T14:54:07.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T14:54:07.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T14:54:07.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T14:54:07.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T14:54:07.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T14:54:07.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:54:07.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:54:07.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T14:54:07.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:53:47.214999+0000 '>' 2024-09-18T14:53:47.214999+0000 2024-09-18T14:54:07.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T14:54:08.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T14:54:08.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T14:54:08.720 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T14:54:08.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T14:54:08.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:54:08.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:54:08.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T14:54:09.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:53:47.214999+0000 '>' 2024-09-18T14:53:47.214999+0000 2024-09-18T14:54:09.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T14:54:10.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T14:54:10.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T14:54:10.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T14:54:10.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T14:54:10.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:54:10.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:54:10.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T14:54:10.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:53:47.214999+0000 '>' 2024-09-18T14:53:47.214999+0000 2024-09-18T14:54:10.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T14:54:11.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T14:54:11.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T14:54:11.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T14:54:11.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T14:54:11.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:54:11.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:54:11.347 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T14:54:11.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:53:47.214999+0000 '>' 2024-09-18T14:53:47.214999+0000 2024-09-18T14:54:11.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T14:54:12.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T14:54:12.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T14:54:12.666 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T14:54:12.666 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T14:54:12.667 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:54:12.667 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:54:12.667 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T14:54:12.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:54:08.927755+0000 '>' 2024-09-18T14:53:47.214999+0000 2024-09-18T14:54:12.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T14:54:12.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-09-18T14:54:12.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:54:12.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:54:12.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:54:12.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T14:54:12.974 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:54:12.974 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:54:12.974 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:54:12.974 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:54:12.975 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:54:12.975 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:54:13.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:54:13.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:54:13.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:54:13.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:54:13.192 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:54:13.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:54:13.504 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:54:13.504 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T14:54:13.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:54:13.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:54:13.505 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:54:13.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182405 2024-09-18T14:54:13.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182405 2024-09-18T14:54:13.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182405' 2024-09-18T14:54:13.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:54:13.675 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:54:13.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672979 2024-09-18T14:54:13.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672979 2024-09-18T14:54:13.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182405 1-42949672979' 2024-09-18T14:54:13.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:54:13.818 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:54:13.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509453 2024-09-18T14:54:13.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509453 2024-09-18T14:54:13.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182405 1-42949672979 2-64424509453' 2024-09-18T14:54:13.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:54:13.956 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-107374182405 2024-09-18T14:54:13.957 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:54:13.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:54:13.960 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-107374182405 2024-09-18T14:54:13.960 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:54:13.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182405 2024-09-18T14:54:13.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 107374182405' 2024-09-18T14:54:13.962 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 107374182405 2024-09-18T14:54:13.962 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:54:14.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182404 -lt 107374182405 2024-09-18T14:54:14.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:54:15.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:54:15.267 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:54:15.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182405 -lt 107374182405 2024-09-18T14:54:15.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:54:15.633 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672979 2024-09-18T14:54:15.633 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:54:15.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:54:15.635 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672979 2024-09-18T14:54:15.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:54:15.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672979 2024-09-18T14:54:15.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-18T14:54:15.638 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672979 2024-09-18T14:54:15.638 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:54:15.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672979 -lt 42949672979 2024-09-18T14:54:15.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:54:15.980 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509453 2024-09-18T14:54:15.980 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:54:15.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:54:15.984 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509453 2024-09-18T14:54:15.984 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:54:15.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509453 2024-09-18T14:54:15.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-18T14:54:15.985 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509453 2024-09-18T14:54:15.986 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:54:16.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509453 -lt 64424509453 2024-09-18T14:54:16.329 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:54:16.329 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:16.330 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:16.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T14:54:16.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:16.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:16.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:16.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:16.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:16.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:16.744 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:17.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T14:54:17.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:17.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:17.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:17.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T14:54:17.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:54:17.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:54:17.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: ceph pg dump pgs 2024-09-18T14:54:17.773 INFO:tasks.workunit.client.0.smithi046.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:54:17.773 INFO:tasks.workunit.client.0.smithi046.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-09-18T14:54:09.084000+0000 21'1 26:62 [1,0] 1 [1,0] 1 21'1 2024-09-18T14:54:08.927755+0000 21'1 2024-09-18T14:54:08.927755+0000 0 1 periodic scrub scheduled @ 2024-09-19T14:54:08.927755+0000 1 0 2024-09-18T14:54:17.774 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-09-18T14:54:17.774 INFO:tasks.workunit.client.0.smithi046.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:54:17.777 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:54:17.786 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:480: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-09-18T14:54:17.786 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-09-18T14:54:17.787 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-09-18T14:54:17.787 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-09-18T14:54:17.787 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-09-18T14:54:17.787 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T14:54:17.788 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T14:54:17.788 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T14:54:18.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T14:54:18.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T14:54:18.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T14:54:18.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:480: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-18T14:54:18.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T14:54:18.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T14:54:18.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T14:54:18.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T14:54:18.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-18T14:54:18.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T14:54:18.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T14:54:18.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T14:54:18.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T14:54:18.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T14:54:18.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:54:18.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:54:18.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:54:18.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:54:18.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:54:18.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:54:18.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-18T14:54:18.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T14:54:18.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T14:54:18.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T14:54:18.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T14:54:18.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T14:54:18.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-09-18T14:54:20.262 INFO:tasks.workunit.client.0.smithi046.stdout:_ 2024-09-18T14:54:20.262 INFO:tasks.workunit.client.0.smithi046.stdout:snapset 2024-09-18T14:54:20.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:54:20.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T14:54:20.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:54:20.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T14:54:20.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:54:20.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T14:54:20.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T14:54:20.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:54:20.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:54:20.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:54:20.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T14:54:20.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T14:54:20.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:54:20.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:54:20.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T14:54:20.799 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:54:20.799 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:54:20.799 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:54:20.801 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:54:20.801 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:54:20.801 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:54:20.801 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:54:20.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:54:20.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:54:20.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T14:54:20.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T14:54:20.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:54:20.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:54:20.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:54:20.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:54:20.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:54:20.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T14:54:20.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T14:54:20.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T14:54:20.803 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T14:54:20.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:54:20.804 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T14:54:20.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T14:54:20.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:54:20.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:54:20.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:54:20.836 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:54:20.831+0000 7fc5d0b15540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:54:20.841 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:54:20.835+0000 7fc5d0b15540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:54:20.842 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:54:20.835+0000 7fc5d0b15540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:54:21.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T14:54:21.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:54:21.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:54:21.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:54:21.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:54:21.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:54:21.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:54:21.117 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T14:54:21.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:54:21.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:54:21.381 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:54:21.375+0000 7fc5d0b15540 -1 Falling back to public interface 2024-09-18T14:54:21.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:54:22.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:54:22.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:54:22.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:54:22.464 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:54:22.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:54:22.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:54:22.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:54:23.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:54:23.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:54:23.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:54:23.815 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T14:54:23.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:54:23.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:54:24.071 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:54:24.067+0000 7fc5d0b15540 -1 osd.0 26 log_to_monitors true 2024-09-18T14:54:24.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:54:25.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:54:25.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:54:25.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:54:25.136 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T14:54:25.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:54:25.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:54:25.491 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/153230,v1:127.0.0.1:6803/153230] [v2:127.0.0.1:6804/153230,v1:127.0.0.1:6805/153230] exists,up 49025be4-3064-415c-9430-495d941953eb 2024-09-18T14:54:25.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:54:25.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:54:25.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:54:25.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T14:54:25.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:54:25.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:54:25.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:54:25.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T14:54:25.492 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:54:25.493 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:54:25.493 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:54:25.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:54:25.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:54:25.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:54:25.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:54:25.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:54:25.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:54:25.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:54:25.704 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:54:26.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:54:26.012 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:54:26.012 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T14:54:26.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:54:26.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:54:26.012 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:54:26.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018882 2024-09-18T14:54:26.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018882 2024-09-18T14:54:26.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-128849018882' 2024-09-18T14:54:26.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:54:26.194 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:54:26.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672982 2024-09-18T14:54:26.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672982 2024-09-18T14:54:26.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-128849018882 1-42949672982' 2024-09-18T14:54:26.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:54:26.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:54:26.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509456 2024-09-18T14:54:26.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509456 2024-09-18T14:54:26.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-128849018882 1-42949672982 2-64424509456' 2024-09-18T14:54:26.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:54:26.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-128849018882 2024-09-18T14:54:26.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:54:26.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:54:26.485 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-128849018882 2024-09-18T14:54:26.485 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:54:26.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018882 2024-09-18T14:54:26.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-09-18T14:54:26.488 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 128849018882 2024-09-18T14:54:26.488 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:54:26.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018882 -lt 128849018882 2024-09-18T14:54:26.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:54:26.826 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672982 2024-09-18T14:54:26.827 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:54:26.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:54:26.830 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672982 2024-09-18T14:54:26.830 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:54:26.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672982 2024-09-18T14:54:26.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-09-18T14:54:26.832 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 42949672982 2024-09-18T14:54:26.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:54:27.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672983 -lt 42949672982 2024-09-18T14:54:27.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:54:27.166 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509456 2024-09-18T14:54:27.166 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:54:27.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:54:27.169 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509456 2024-09-18T14:54:27.169 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:54:27.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509456 2024-09-18T14:54:27.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-09-18T14:54:27.170 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509456 2024-09-18T14:54:27.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:54:27.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509456 -lt 64424509456 2024-09-18T14:54:27.633 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:54:27.633 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:27.634 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:28.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T14:54:28.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:28.048 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:28.048 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:28.048 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:28.048 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:28.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:28.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:28.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T14:54:28.401 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:28.401 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:28.402 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:28.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T14:54:28.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:54:28.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:54:28.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:481: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-09-18T14:54:28.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-09-18T14:54:28.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-09-18T14:54:28.766 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-09-18T14:54:28.766 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-09-18T14:54:28.766 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T14:54:28.767 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T14:54:28.767 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T14:54:29.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T14:54:29.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T14:54:29.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T14:54:29.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:481: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-18T14:54:29.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T14:54:29.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T14:54:29.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T14:54:29.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T14:54:29.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-18T14:54:29.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T14:54:29.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T14:54:29.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T14:54:29.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T14:54:29.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T14:54:29.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:54:29.476 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:54:29.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:54:29.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:54:29.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:54:29.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:54:29.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-18T14:54:29.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T14:54:29.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T14:54:29.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T14:54:29.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T14:54:29.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T14:54:29.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-18T14:54:31.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:54:31.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T14:54:31.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:54:31.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T14:54:31.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:54:31.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T14:54:31.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T14:54:31.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:54:31.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:54:31.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:54:31.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T14:54:31.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T14:54:31.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:54:31.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:54:31.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T14:54:31.800 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:54:31.800 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:54:31.801 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:54:31.801 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:54:31.801 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:54:31.801 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:54:31.802 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:54:31.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:54:31.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:54:31.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T14:54:31.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T14:54:31.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:54:31.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:54:31.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:54:31.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:54:31.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:54:31.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T14:54:31.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T14:54:31.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T14:54:31.804 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T14:54:31.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:54:31.804 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T14:54:31.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T14:54:31.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:54:31.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:54:31.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:54:31.836 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:54:31.831+0000 7f59edd54540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:54:31.839 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:54:31.831+0000 7f59edd54540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:54:31.841 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:54:31.835+0000 7f59edd54540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:54:32.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T14:54:32.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:54:32.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:54:32.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:54:32.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:54:32.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:54:32.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:54:32.122 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T14:54:32.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:54:32.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:54:32.384 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:54:32.375+0000 7f59edd54540 -1 Falling back to public interface 2024-09-18T14:54:32.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:54:33.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:54:33.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:54:33.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:54:33.464 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:54:33.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:54:33.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:54:33.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:54:34.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:54:34.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:54:34.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:54:34.777 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T14:54:34.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:54:34.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:54:35.045 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:54:35.039+0000 7f59edd54540 -1 osd.0 31 log_to_monitors true 2024-09-18T14:54:35.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:54:36.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:54:36.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:54:36.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:54:36.099 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T14:54:36.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:54:36.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:54:36.459 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/154610,v1:127.0.0.1:6803/154610] [v2:127.0.0.1:6804/154610,v1:127.0.0.1:6805/154610] exists,up 49025be4-3064-415c-9430-495d941953eb 2024-09-18T14:54:36.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:54:36.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:54:36.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:54:36.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T14:54:36.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:54:36.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:54:36.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:54:36.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T14:54:36.460 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:54:36.460 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:54:36.460 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:54:36.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:54:36.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:54:36.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:54:36.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:54:36.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:54:36.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:54:36.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:54:36.677 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:54:37.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:54:37.001 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:54:37.001 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T14:54:37.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:54:37.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:54:37.001 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:54:37.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855362 2024-09-18T14:54:37.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855362 2024-09-18T14:54:37.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855362' 2024-09-18T14:54:37.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:54:37.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:54:37.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672986 2024-09-18T14:54:37.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672986 2024-09-18T14:54:37.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855362 1-42949672986' 2024-09-18T14:54:37.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:54:37.322 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:54:37.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509460 2024-09-18T14:54:37.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509460 2024-09-18T14:54:37.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855362 1-42949672986 2-64424509460' 2024-09-18T14:54:37.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:54:37.471 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-150323855362 2024-09-18T14:54:37.471 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:54:37.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:54:37.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-150323855362 2024-09-18T14:54:37.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:54:37.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855362 2024-09-18T14:54:37.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-09-18T14:54:37.476 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 150323855362 2024-09-18T14:54:37.477 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:54:37.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 150323855362 2024-09-18T14:54:37.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:54:38.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:54:38.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:54:39.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855362 -lt 150323855362 2024-09-18T14:54:39.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:54:39.108 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672986 2024-09-18T14:54:39.108 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:54:39.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:54:39.112 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672986 2024-09-18T14:54:39.112 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:54:39.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672986 2024-09-18T14:54:39.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-09-18T14:54:39.113 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 42949672986 2024-09-18T14:54:39.114 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:54:39.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672986 -lt 42949672986 2024-09-18T14:54:39.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:54:39.469 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509460 2024-09-18T14:54:39.469 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:54:39.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:54:39.472 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509460 2024-09-18T14:54:39.473 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:54:39.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509460 2024-09-18T14:54:39.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-09-18T14:54:39.474 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509460 2024-09-18T14:54:39.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:54:39.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509460 -lt 64424509460 2024-09-18T14:54:39.781 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:54:39.781 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:39.782 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:40.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T14:54:40.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:40.198 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:40.198 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:40.198 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:40.198 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:40.198 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:40.198 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:40.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T14:54:40.569 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:40.570 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:40.570 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:40.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T14:54:40.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:54:40.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:54:40.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:482: TEST_auto_repair_bluestore_basic: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-18T14:54:40.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:483: TEST_auto_repair_bluestore_basic: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-18T14:54:40.936 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:54:08.923+0000 7f87f7ea3700 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber : scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-18T14:54:40.936 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:54:08.923+0000 7f87f7ea3700 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber : _scrub_finish info stats: valid 2024-09-18T14:54:40.936 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:54:08.923+0000 7f87f7ea3700 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber : scrub_finish All may be fixed 2024-09-18T14:54:40.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T14:54:40.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T14:54:40.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:54:40.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T14:54:40.937 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:54:40.938 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:54:40.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:54:40.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:54:40.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:54:41.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:54:41.056 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:54:41.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:54:41.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:54:41.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:54:41.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:54:41.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:54:41.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:54:41.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:54:41.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:54:41.064 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:54:41.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:54:41.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:54:41.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T14:54:41.089 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:54:41.089 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:54:41.089 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:54:41.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T14:54:41.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:54:41.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:54:41.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T14:54:41.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T14:54:41.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T14:54:41.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T14:54:41.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T14:54:41.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:54:41.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T14:54:41.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:54:41.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:54:41.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:54:41.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:54:41.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:54:41.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:54:41.098 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:54:41.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:54:41.100 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:54:41.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:54:41.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:54:41.103 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:54:41.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:54:41.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:54:41.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:54:41.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:54:41.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:54:41.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:54:41.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T14:54:41.110 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:54:41.110 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:54:41.110 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:54:41.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T14:54:41.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:54:41.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:54:41.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T14:54:41.115 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:54:41.115 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:54:41.115 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:54:41.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T14:54:41.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:54:41.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T14:54:41.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:54:41.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T14:54:41.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_auto_repair_bluestore_failed td/osd-scrub-repair 2024-09-18T14:54:41.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:539: TEST_auto_repair_bluestore_failed: local dir=td/osd-scrub-repair 2024-09-18T14:54:41.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:540: TEST_auto_repair_bluestore_failed: local poolname=testpool 2024-09-18T14:54:41.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:543: TEST_auto_repair_bluestore_failed: run_mon td/osd-scrub-repair a 2024-09-18T14:54:41.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T14:54:41.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:54:41.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:54:41.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:54:41.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T14:54:41.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-18T14:54:41.502 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:54:41.502 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:54:41.502 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:54:41.503 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:54:41.503 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:54:41.503 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:54:41.503 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:54:41.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T14:54:41.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:54:41.818 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:54:41.818 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:54:41.818 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:54:41.818 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:54:41.818 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:54:41.819 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:54:41.819 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:54:41.819 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:54:41.820 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:54:41.820 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:54:41.820 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:54:41.821 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T14:54:41.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:54:41.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T14:54:41.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:54:41.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:54:41.914 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:54:41.914 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:54:41.914 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:54:41.915 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:54:41.915 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:54:41.915 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:54:41.915 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:54:41.916 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:54:41.916 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:54:41.916 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T14:54:41.917 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:54:41.917 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T14:54:42.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:544: TEST_auto_repair_bluestore_failed: run_mgr td/osd-scrub-repair x 2024-09-18T14:54:42.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T14:54:42.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:54:42.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:54:42.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:54:42.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T14:54:42.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:54:42.862 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:54:42.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:54:42.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:54:42.863 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:54:42.863 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:54:42.863 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:54:42.864 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:54:42.864 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:54:42.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:54:42.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:547: TEST_auto_repair_bluestore_failed: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T14:54:42.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: seq 0 2 2024-09-18T14:54:42.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-09-18T14:54:42.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:54:42.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T14:54:42.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:54:42.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:54:42.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:54:42.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T14:54:42.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T14:54:42.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:54:42.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:54:42.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:54:42.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T14:54:42.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T14:54:42.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:54:42.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:54:42.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T14:54:42.907 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:54:42.907 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:54:42.907 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:54:42.907 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:54:42.908 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:54:42.908 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:54:42.909 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:54:42.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:54:42.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:54:42.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:54:42.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:54:42.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T14:54:42.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T14:54:42.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:54:42.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:54:42.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:54:42.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:54:42.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:54:42.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T14:54:42.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T14:54:42.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:54:42.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=84578cdf-ea8c-423e-916f-6ca29549de31 2024-09-18T14:54:42.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 84578cdf-ea8c-423e-916f-6ca29549de31' 2024-09-18T14:54:42.911 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 84578cdf-ea8c-423e-916f-6ca29549de31 2024-09-18T14:54:42.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:54:42.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCy6epmPFwiNxAALjzI+hrBZTbZDoFHI9dVaw== 2024-09-18T14:54:42.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCy6epmPFwiNxAALjzI+hrBZTbZDoFHI9dVaw=="}' 2024-09-18T14:54:42.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 84578cdf-ea8c-423e-916f-6ca29549de31 -i td/osd-scrub-repair/0/new.json 2024-09-18T14:54:43.210 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:54:43.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-18T14:54:43.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCy6epmPFwiNxAALjzI+hrBZTbZDoFHI9dVaw== --osd-uuid 84578cdf-ea8c-423e-916f-6ca29549de31 2024-09-18T14:54:43.261 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:54:43.255+0000 7fab3dfa8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:54:43.304 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:54:43.299+0000 7fab3dfa8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:54:43.306 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:54:43.299+0000 7fab3dfa8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:54:43.307 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:54:43.299+0000 7fab3dfa8540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T14:54:43.307 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:54:43.299+0000 7fab3dfa8540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T14:54:43.307 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:54:43.299+0000 7fab3dfa8540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T14:54:43.355 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:54:43.347+0000 7fab3dfa8540 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-18T14:54:49.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T14:54:49.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:54:49.474 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T14:54:49.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:54:49.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:54:49.869 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T14:54:49.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:54:49.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:54:49.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:54:49.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:54:49.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:54:49.901 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:54:49.895+0000 7f0c8f77e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:54:49.904 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:54:49.899+0000 7f0c8f77e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:54:49.905 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:54:49.899+0000 7f0c8f77e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:54:50.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:54:50.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:54:50.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:54:50.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:54:50.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:54:50.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:54:50.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:54:50.203 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:54:50.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:54:50.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:54:50.438 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:54:50.431+0000 7f0c8f77e540 -1 Falling back to public interface 2024-09-18T14:54:50.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:54:51.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:54:51.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:54:51.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:54:51.551 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:54:51.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:54:51.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:54:51.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:54:52.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:54:52.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:54:52.867 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:54:52.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:54:52.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:54:52.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:54:53.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:54:53.207 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:54:53.203+0000 7f0c8f77e540 -1 osd.0 0 log_to_monitors true 2024-09-18T14:54:54.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:54:54.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:54:54.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:54:54.182 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:54:54.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:54:54.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:54:54.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:54:55.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:54:55.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:54:55.504 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:54:55.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:54:55.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:54:55.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:54:55.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:54:56.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:54:56.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:54:56.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:54:56.819 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:54:56.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:54:56.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:54:57.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:54:58.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:54:58.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:54:58.217 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:54:58.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:54:58.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:54:58.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:54:58.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:54:59.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:54:59.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:54:59.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:54:59.534 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:54:59.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:54:59.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:54:59.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:00.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:00.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:00.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:55:00.842 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T14:55:00.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:00.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:55:01.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:02.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:02.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:02.241 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T14:55:02.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:55:02.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:02.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:55:02.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:03.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:03.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:03.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:55:03.561 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T14:55:03.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:03.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:55:03.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:04.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:04.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:04.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:55:04.983 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T14:55:04.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:04.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:55:05.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:06.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:06.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:06.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:55:06.296 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T14:55:06.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:06.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:55:07.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:08.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:08.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:08.008 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T14:55:08.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:55:08.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:08.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:55:08.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:09.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:09.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:09.459 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T14:55:09.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:55:09.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:09.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:55:10.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:11.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:11.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:11.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:55:11.313 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T14:55:11.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:11.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:55:12.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:13.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:13.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:13.148 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T14:55:13.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:55:13.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:13.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:55:13.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:14.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:14.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:14.919 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T14:55:14.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:55:14.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:14.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:55:15.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:16.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:16.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:16.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:55:16.507 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T14:55:16.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:16.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:55:16.823 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/156703,v1:127.0.0.1:6803/156703] [v2:127.0.0.1:6804/156703,v1:127.0.0.1:6805/156703] exists,up 84578cdf-ea8c-423e-916f-6ca29549de31 2024-09-18T14:55:16.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:55:16.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:55:16.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:55:16.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-09-18T14:55:16.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:55:16.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T14:55:16.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:55:16.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T14:55:16.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:55:16.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T14:55:16.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T14:55:16.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:55:16.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:55:16.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:55:16.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T14:55:16.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T14:55:16.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:55:16.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:55:16.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T14:55:16.827 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:55:16.827 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:55:16.827 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:55:16.829 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:55:16.829 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:55:16.829 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:55:16.829 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:55:16.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:55:16.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:55:16.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:55:16.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:55:16.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T14:55:16.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T14:55:16.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:55:16.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:55:16.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:55:16.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:55:16.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:55:16.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T14:55:16.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T14:55:16.830 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:55:16.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e505ad50-da22-495a-91f7-4ce2d5176fe2 2024-09-18T14:55:16.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e505ad50-da22-495a-91f7-4ce2d5176fe2' 2024-09-18T14:55:16.831 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 e505ad50-da22-495a-91f7-4ce2d5176fe2 2024-09-18T14:55:16.832 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:55:16.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDU6epm0a+OMhAALXnC5um6TnvCCSJ8vGjkbg== 2024-09-18T14:55:16.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDU6epm0a+OMhAALXnC5um6TnvCCSJ8vGjkbg=="}' 2024-09-18T14:55:16.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e505ad50-da22-495a-91f7-4ce2d5176fe2 -i td/osd-scrub-repair/1/new.json 2024-09-18T14:55:17.261 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:55:17.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-18T14:55:17.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDU6epm0a+OMhAALXnC5um6TnvCCSJ8vGjkbg== --osd-uuid e505ad50-da22-495a-91f7-4ce2d5176fe2 2024-09-18T14:55:17.318 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:55:17.310+0000 7f2bae113540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:55:17.321 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:55:17.314+0000 7f2bae113540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:55:17.323 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:55:17.314+0000 7f2bae113540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:55:17.323 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:55:17.318+0000 7f2bae113540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T14:55:17.323 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:55:17.318+0000 7f2bae113540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T14:55:17.323 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:55:17.318+0000 7f2bae113540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T14:55:17.385 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:55:17.378+0000 7f2bae113540 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-18T14:55:23.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-18T14:55:23.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:55:23.866 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T14:55:23.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T14:55:23.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:55:24.258 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T14:55:24.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T14:55:24.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:55:24.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:55:24.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:55:24.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:55:24.291 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:55:24.286+0000 7f9c27337540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:55:24.306 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:55:24.298+0000 7f9c27337540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:55:24.307 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:55:24.302+0000 7f9c27337540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:55:24.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T14:55:24.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:55:24.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:55:24.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:55:24.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:55:24.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:24.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:55:24.577 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:55:24.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:24.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:55:24.861 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:55:24.854+0000 7f9c27337540 -1 Falling back to public interface 2024-09-18T14:55:24.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:25.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:25.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:25.881 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:55:25.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:55:25.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:25.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:55:26.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:27.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:27.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:27.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:55:27.197 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:55:27.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:27.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:55:27.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:27.770 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:55:27.762+0000 7f9c27337540 -1 osd.1 0 log_to_monitors true 2024-09-18T14:55:28.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:28.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:28.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:55:28.513 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:55:28.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:28.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:55:28.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:29.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:29.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:29.935 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:55:29.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:55:29.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:29.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:55:30.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:31.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:31.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:31.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:55:31.249 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:55:31.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:31.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:55:31.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:32.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:32.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:32.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:55:32.665 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:55:32.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:32.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:55:33.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:34.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:34.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:34.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:55:34.057 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:55:34.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:34.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:55:34.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:35.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:35.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:35.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:55:35.382 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T14:55:35.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:35.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:55:35.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:36.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:36.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:36.814 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T14:55:36.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:55:36.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:36.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:55:37.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:38.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:38.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:38.252 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T14:55:38.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:55:38.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:38.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:55:38.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:39.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:39.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:39.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:55:39.738 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T14:55:39.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:39.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:55:40.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:41.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:41.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:41.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:55:41.041 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T14:55:41.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:41.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:55:41.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:42.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:42.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:42.850 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T14:55:42.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:55:42.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:42.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:55:43.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:44.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:44.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:44.592 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T14:55:44.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:55:44.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:44.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:55:45.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:46.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:46.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:46.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:55:46.006 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T14:55:46.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:46.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:55:46.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:47.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:47.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:47.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:55:47.484 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T14:55:47.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:47.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:55:47.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:48.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:48.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:48.961 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T14:55:48.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:55:48.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:55:48.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:49.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:50.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:50.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:50.394 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T14:55:50.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:55:50.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:50.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:55:50.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:51.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:51.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:51.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T14:55:51.987 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T14:55:51.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:51.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:55:52.308 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/158486,v1:127.0.0.1:6811/158486] [v2:127.0.0.1:6812/158486,v1:127.0.0.1:6813/158486] exists,up e505ad50-da22-495a-91f7-4ce2d5176fe2 2024-09-18T14:55:52.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:55:52.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:55:52.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:55:52.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-09-18T14:55:52.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:55:52.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T14:55:52.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:55:52.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T14:55:52.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:55:52.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-18T14:55:52.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T14:55:52.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:55:52.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:55:52.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:55:52.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T14:55:52.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T14:55:52.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:55:52.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:55:52.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T14:55:52.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:55:52.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:55:52.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:55:52.313 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:55:52.314 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:55:52.314 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:55:52.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:55:52.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:55:52.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:55:52.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:55:52.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:55:52.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T14:55:52.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T14:55:52.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:55:52.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:55:52.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:55:52.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:55:52.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:55:52.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T14:55:52.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-18T14:55:52.315 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:55:52.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=26056d9e-9b36-4d94-84e0-31a916b35972 2024-09-18T14:55:52.316 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 26056d9e-9b36-4d94-84e0-31a916b35972 2024-09-18T14:55:52.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 26056d9e-9b36-4d94-84e0-31a916b35972' 2024-09-18T14:55:52.317 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:55:52.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD46epmP6OrExAAeRf0K5bRVQuYKAeCYidNgQ== 2024-09-18T14:55:52.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD46epmP6OrExAAeRf0K5bRVQuYKAeCYidNgQ=="}' 2024-09-18T14:55:52.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 26056d9e-9b36-4d94-84e0-31a916b35972 -i td/osd-scrub-repair/2/new.json 2024-09-18T14:55:52.756 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:55:52.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-18T14:55:52.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQD46epmP6OrExAAeRf0K5bRVQuYKAeCYidNgQ== --osd-uuid 26056d9e-9b36-4d94-84e0-31a916b35972 2024-09-18T14:55:52.810 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:55:52.802+0000 7f7ffc2aa540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:55:52.815 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:55:52.810+0000 7f7ffc2aa540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:55:52.817 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:55:52.810+0000 7f7ffc2aa540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:55:52.818 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:55:52.810+0000 7f7ffc2aa540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T14:55:52.818 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:55:52.810+0000 7f7ffc2aa540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T14:55:52.819 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:55:52.810+0000 7f7ffc2aa540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T14:55:52.882 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:55:52.874+0000 7f7ffc2aa540 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-18T14:55:59.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-18T14:55:59.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:55:59.126 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T14:55:59.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T14:55:59.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:55:59.528 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T14:55:59.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T14:55:59.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:55:59.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:55:59.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:55:59.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:55:59.563 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:55:59.554+0000 7f04ef62d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:55:59.565 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:55:59.558+0000 7f04ef62d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:55:59.567 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:55:59.558+0000 7f04ef62d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:55:59.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T14:55:59.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:55:59.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:55:59.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:55:59.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:55:59.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:59.872 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:55:59.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:55:59.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:59.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:56:00.102 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:56:00.094+0000 7f04ef62d540 -1 Falling back to public interface 2024-09-18T14:56:00.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:01.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:01.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:01.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:56:01.357 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:56:01.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:01.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:56:01.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:02.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:02.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:02.674 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:56:02.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:56:02.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:02.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:56:02.843 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:56:02.834+0000 7f04ef62d540 -1 osd.2 0 log_to_monitors true 2024-09-18T14:56:02.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:03.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:03.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:03.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:56:03.991 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:56:03.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:03.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:56:04.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:05.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:05.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:05.366 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:56:05.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:56:05.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:05.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:56:05.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:06.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:06.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:06.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:56:06.711 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:56:06.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:06.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:56:07.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:08.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:08.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:08.051 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:56:08.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:56:08.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:08.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:56:08.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:09.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:09.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:09.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:56:09.414 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:56:09.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:09.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:56:09.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:10.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:10.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:10.720 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T14:56:10.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:56:10.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:10.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:56:11.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:12.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:12.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:12.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:56:12.072 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T14:56:12.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:12.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:56:12.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:13.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:13.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:13.389 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T14:56:13.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:56:13.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:13.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:56:13.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:14.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:14.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:14.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:56:14.728 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T14:56:14.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:14.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:56:15.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:16.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:16.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:16.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:56:16.053 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T14:56:16.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:16.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:56:16.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:17.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:17.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:17.911 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T14:56:17.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:56:17.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:17.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:56:18.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:19.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:19.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:19.670 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T14:56:19.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:56:19.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:19.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:56:20.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:21.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:21.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:21.514 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T14:56:21.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:56:21.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:21.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:56:22.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:23.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:23.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:23.277 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T14:56:23.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:56:23.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:23.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:56:23.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:24.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:24.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:24.749 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T14:56:24.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:56:24.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:24.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:56:25.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:26.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:26.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:26.435 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T14:56:26.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:56:26.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:26.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:56:26.827 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/160290,v1:127.0.0.1:6819/160290] [v2:127.0.0.1:6820/160290,v1:127.0.0.1:6821/160290] exists,up 26056d9e-9b36-4d94-84e0-31a916b35972 2024-09-18T14:56:26.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:56:26.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:56:26.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:56:26.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_failed: create_pool testpool 1 1 2024-09-18T14:56:26.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-09-18T14:56:27.316 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'testpool' created 2024-09-18T14:56:27.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:56:28.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_failed: ceph osd pool set testpool size 2 2024-09-18T14:56:29.074 INFO:tasks.workunit.client.0.smithi046.stderr:set pool 1 size to 2 2024-09-18T14:56:29.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:554: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-09-18T14:56:29.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:56:29.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:56:29.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:56:29.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T14:56:29.088 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:56:29.088 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:56:29.089 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:56:29.089 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:56:29.089 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:56:29.089 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:56:29.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:56:29.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:56:29.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:56:29.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:56:29.305 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:56:29.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:56:29.694 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:56:29.694 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T14:56:29.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:56:29.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:56:29.694 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:56:29.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836501 2024-09-18T14:56:29.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836501 2024-09-18T14:56:29.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836501' 2024-09-18T14:56:29.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:56:29.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:56:29.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=38654705679 2024-09-18T14:56:29.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 38654705679 2024-09-18T14:56:29.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836501 1-38654705679' 2024-09-18T14:56:29.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:56:29.973 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:56:30.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=60129542154 2024-09-18T14:56:30.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 60129542154 2024-09-18T14:56:30.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836501 1-38654705679 2-60129542154' 2024-09-18T14:56:30.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:56:30.117 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836501 2024-09-18T14:56:30.117 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:56:30.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:56:30.119 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836501 2024-09-18T14:56:30.120 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:56:30.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836501 2024-09-18T14:56:30.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-18T14:56:30.122 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836501 2024-09-18T14:56:30.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:56:30.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836501 -lt 21474836501 2024-09-18T14:56:30.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:56:30.430 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-38654705679 2024-09-18T14:56:30.431 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:56:30.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:56:30.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-38654705679 2024-09-18T14:56:30.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:56:30.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=38654705679 2024-09-18T14:56:30.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 38654705679' 2024-09-18T14:56:30.435 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 38654705679 2024-09-18T14:56:30.436 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:56:30.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 38654705679 -lt 38654705679 2024-09-18T14:56:30.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:56:30.744 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-60129542154 2024-09-18T14:56:30.744 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:56:30.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:56:30.747 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-60129542154 2024-09-18T14:56:30.748 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:56:30.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=60129542154 2024-09-18T14:56:30.750 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 60129542154 2024-09-18T14:56:30.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 60129542154' 2024-09-18T14:56:30.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:56:31.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 60129542154 -lt 60129542154 2024-09-18T14:56:31.128 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:56:31.129 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:56:31.129 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:56:31.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T14:56:31.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:56:31.521 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:56:31.521 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:56:31.521 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:56:31.521 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:56:31.521 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:56:31.521 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:56:31.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:56:31.864 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:56:31.864 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:56:31.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:56:32.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:56:32.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-18T14:56:32.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T14:56:32.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-18T14:56:32.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:56:32.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:56:32.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:56:32.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:56:32.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:56:32.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:56:32.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:56:32.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:56:32.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:56:32.347 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:56:32.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:56:32.654 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:56:32.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:56:32.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:56:33.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:56:33.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:56:33.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:56:33.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:56:33.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:56:33.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:56:33.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:56:33.024 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:56:33.024 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:56:33.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:56:33.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:56:33.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-09-18T14:56:33.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:56:33.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-18T14:56:33.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:56:33.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:56:33.588 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:56:33.589 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:56:33.589 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:56:33.589 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:56:33.589 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:56:33.589 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:56:33.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:56:33.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:56:33.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:56:33.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:56:34.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:56:34.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:56:34.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:56:34.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:56:34.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:56:34.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:56:34.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:56:34.269 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:56:34.269 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:56:34.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:56:34.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:56:34.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-09-18T14:56:34.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:56:34.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-18T14:56:35.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:56:35.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:56:35.038 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:56:35.038 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:56:35.038 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:56:35.038 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:56:35.039 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:56:35.039 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:56:35.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T14:56:35.354 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:56:35.354 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:56:35.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:56:35.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T14:56:35.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:56:35.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:56:35.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: local payload=ABCDEF 2024-09-18T14:56:35.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:558: TEST_auto_repair_bluestore_failed: echo ABCDEF 2024-09-18T14:56:35.724 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: seq 1 10 2024-09-18T14:56:35.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-18T14:56:35.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-09-18T14:56:35.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-18T14:56:35.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-09-18T14:56:36.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-18T14:56:36.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-09-18T14:56:36.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-18T14:56:36.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-09-18T14:56:36.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-18T14:56:36.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-09-18T14:56:36.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-18T14:56:36.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-09-18T14:56:36.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-18T14:56:36.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-09-18T14:56:36.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-18T14:56:36.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-09-18T14:56:36.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-18T14:56:36.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-09-18T14:56:37.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-18T14:56:37.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-09-18T14:56:37.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-09-18T14:56:37.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-09-18T14:56:37.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-09-18T14:56:37.280 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-09-18T14:56:37.280 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-09-18T14:56:37.281 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T14:56:37.281 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T14:56:37.281 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T14:56:37.595 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T14:56:37.596 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T14:56:37.596 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T14:56:37.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-09-18T14:56:37.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T14:56:37.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T14:56:37.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T14:56:37.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T14:56:37.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-09-18T14:56:37.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T14:56:37.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T14:56:37.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T14:56:37.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T14:56:37.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T14:56:37.918 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:56:37.918 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:56:37.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:56:37.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:56:37.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:56:38.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:56:38.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-09-18T14:56:38.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T14:56:38.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T14:56:38.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T14:56:38.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T14:56:38.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T14:56:38.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-09-18T14:56:39.878 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:ff7b1f36:::obj1:head# 2024-09-18T14:56:40.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:56:40.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T14:56:40.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:56:40.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T14:56:40.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:56:40.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T14:56:40.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T14:56:40.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:56:40.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:56:40.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:56:40.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T14:56:40.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T14:56:40.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:56:40.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:56:40.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T14:56:40.416 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:56:40.416 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:56:40.416 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:56:40.418 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:56:40.418 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:56:40.418 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:56:40.418 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:56:40.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:56:40.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:56:40.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T14:56:40.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T14:56:40.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:56:40.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:56:40.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:56:40.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:56:40.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:56:40.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T14:56:40.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T14:56:40.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T14:56:40.419 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T14:56:40.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:56:40.420 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T14:56:40.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T14:56:40.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:56:40.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:56:40.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:56:40.454 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:56:40.445+0000 7f8b3eb2c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:40.458 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:56:40.449+0000 7f8b3eb2c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:40.463 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:56:40.453+0000 7f8b3eb2c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:40.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T14:56:40.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:56:40.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:56:40.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:56:40.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:56:40.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:40.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:56:40.726 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T14:56:40.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:40.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:56:40.990 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:56:40.981+0000 7f8b3eb2c540 -1 Falling back to public interface 2024-09-18T14:56:41.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:42.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:42.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:42.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:56:42.063 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:56:42.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:42.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:56:42.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:43.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:43.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:43.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:56:43.381 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T14:56:43.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:43.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:56:43.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:43.704 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:56:43.697+0000 7f8b3eb2c540 -1 osd.0 20 log_to_monitors true 2024-09-18T14:56:44.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:44.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:44.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:56:44.704 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T14:56:44.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:44.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:56:45.025 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6802/162498,v1:127.0.0.1:6803/162498] [v2:127.0.0.1:6804/162498,v1:127.0.0.1:6805/162498] exists,up 84578cdf-ea8c-423e-916f-6ca29549de31 2024-09-18T14:56:45.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:56:45.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:56:45.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:56:45.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T14:56:45.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:56:45.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:56:45.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:56:45.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T14:56:45.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:56:45.027 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:56:45.027 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:56:45.027 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:56:45.027 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:56:45.027 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:56:45.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:56:45.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:56:45.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:56:45.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:56:45.238 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:56:45.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:56:45.549 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:56:45.549 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T14:56:45.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:56:45.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:56:45.550 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:56:45.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=103079215106 2024-09-18T14:56:45.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 103079215106 2024-09-18T14:56:45.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-103079215106' 2024-09-18T14:56:45.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:56:45.729 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:56:45.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=38654705683 2024-09-18T14:56:45.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 38654705683 2024-09-18T14:56:45.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-103079215106 1-38654705683' 2024-09-18T14:56:45.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:56:45.874 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:56:46.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=60129542158 2024-09-18T14:56:46.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 60129542158 2024-09-18T14:56:46.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-103079215106 1-38654705683 2-60129542158' 2024-09-18T14:56:46.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:56:46.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-103079215106 2024-09-18T14:56:46.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:56:46.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:56:46.025 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-103079215106 2024-09-18T14:56:46.025 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:56:46.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=103079215106 2024-09-18T14:56:46.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2024-09-18T14:56:46.027 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 103079215106 2024-09-18T14:56:46.028 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:56:46.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 103079215106 2024-09-18T14:56:46.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:56:47.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:56:47.341 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:56:47.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 103079215106 -lt 103079215106 2024-09-18T14:56:47.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:56:47.665 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-38654705683 2024-09-18T14:56:47.665 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:56:47.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:56:47.668 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-38654705683 2024-09-18T14:56:47.668 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:56:47.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=38654705683 2024-09-18T14:56:47.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 38654705683' 2024-09-18T14:56:47.670 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 38654705683 2024-09-18T14:56:47.671 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:56:48.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 38654705683 -lt 38654705683 2024-09-18T14:56:48.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:56:48.004 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-60129542158 2024-09-18T14:56:48.004 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:56:48.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:56:48.007 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-60129542158 2024-09-18T14:56:48.007 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:56:48.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=60129542158 2024-09-18T14:56:48.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 60129542158' 2024-09-18T14:56:48.009 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 60129542158 2024-09-18T14:56:48.010 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:56:48.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 60129542158 -lt 60129542158 2024-09-18T14:56:48.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:56:48.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:56:48.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:56:48.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T14:56:48.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:56:48.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:56:48.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:56:48.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:56:48.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:56:48.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:56:48.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:56:49.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T14:56:49.104 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:56:49.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:56:49.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:56:49.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T14:56:49.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:56:49.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:56:49.471 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-09-18T14:56:49.471 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-09-18T14:56:49.472 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-09-18T14:56:49.472 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-09-18T14:56:49.472 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-09-18T14:56:49.472 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T14:56:49.473 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T14:56:49.473 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T14:56:49.810 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T14:56:49.810 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T14:56:49.811 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T14:56:50.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-09-18T14:56:50.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T14:56:50.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T14:56:50.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T14:56:50.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T14:56:50.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-09-18T14:56:50.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T14:56:50.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T14:56:50.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T14:56:50.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T14:56:50.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T14:56:50.166 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:56:50.166 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:56:50.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:56:50.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:56:50.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:56:50.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:56:50.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-09-18T14:56:50.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T14:56:50.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T14:56:50.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T14:56:50.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T14:56:50.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T14:56:50.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-09-18T14:56:52.018 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:104778fc:::obj2:head# 2024-09-18T14:56:52.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:56:52.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T14:56:52.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:56:52.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T14:56:52.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:56:52.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T14:56:52.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T14:56:52.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:56:52.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:56:52.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:56:52.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T14:56:52.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T14:56:52.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:56:52.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:56:52.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T14:56:52.564 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:56:52.565 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:56:52.565 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:56:52.567 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:56:52.567 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:56:52.567 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:56:52.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:56:52.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:56:52.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:56:52.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T14:56:52.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T14:56:52.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:56:52.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:56:52.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:56:52.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:56:52.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:56:52.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T14:56:52.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T14:56:52.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T14:56:52.569 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T14:56:52.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:56:52.570 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T14:56:52.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T14:56:52.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:56:52.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:56:52.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:56:52.598 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:56:52.589+0000 7fab2dd81540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:52.601 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:56:52.593+0000 7fab2dd81540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:52.603 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:56:52.593+0000 7fab2dd81540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:52.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T14:56:52.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:56:52.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:56:52.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:56:52.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:56:52.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:52.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:56:52.882 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T14:56:52.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:52.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:56:52.894 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:56:52.885+0000 7fab2dd81540 -1 Falling back to public interface 2024-09-18T14:56:53.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:54.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:54.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:54.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:56:54.236 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:56:54.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:54.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:56:54.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:55.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:55.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:55.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:56:55.561 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T14:56:55.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:56:55.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:55.638 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:56:55.629+0000 7fab2dd81540 -1 osd.0 25 log_to_monitors true 2024-09-18T14:56:55.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:56.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:56.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:56.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:56:56.879 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T14:56:56.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:56.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:56:57.286 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 26 last_clean_interval [24,25) [v2:127.0.0.1:6802/163902,v1:127.0.0.1:6803/163902] [v2:127.0.0.1:6804/163902,v1:127.0.0.1:6805/163902] exists,up 84578cdf-ea8c-423e-916f-6ca29549de31 2024-09-18T14:56:57.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:56:57.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:56:57.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:56:57.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T14:56:57.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:56:57.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:56:57.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:56:57.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T14:56:57.287 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:56:57.287 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:56:57.287 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:56:57.288 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:56:57.288 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:56:57.288 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:56:57.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:56:57.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:56:57.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:56:57.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:56:57.501 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:56:57.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:56:57.810 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:56:57.810 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T14:56:57.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:56:57.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:56:57.811 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:56:58.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051586 2024-09-18T14:56:58.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051586 2024-09-18T14:56:58.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051586' 2024-09-18T14:56:58.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:56:58.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:56:58.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=38654705687 2024-09-18T14:56:58.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 38654705687 2024-09-18T14:56:58.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051586 1-38654705687' 2024-09-18T14:56:58.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:56:58.151 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:56:58.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=60129542162 2024-09-18T14:56:58.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 60129542162 2024-09-18T14:56:58.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051586 1-38654705687 2-60129542162' 2024-09-18T14:56:58.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:56:58.290 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-124554051586 2024-09-18T14:56:58.290 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:56:58.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:56:58.293 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-124554051586 2024-09-18T14:56:58.293 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:56:58.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051586 2024-09-18T14:56:58.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-09-18T14:56:58.295 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 124554051586 2024-09-18T14:56:58.296 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:56:58.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051586 -lt 124554051586 2024-09-18T14:56:58.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:56:58.625 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-38654705687 2024-09-18T14:56:58.625 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:56:58.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:56:58.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-38654705687 2024-09-18T14:56:58.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:56:58.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=38654705687 2024-09-18T14:56:58.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 38654705687' 2024-09-18T14:56:58.630 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 38654705687 2024-09-18T14:56:58.631 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:56:59.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 38654705687 -lt 38654705687 2024-09-18T14:56:59.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:56:59.045 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-60129542162 2024-09-18T14:56:59.045 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:56:59.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:56:59.048 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-60129542162 2024-09-18T14:56:59.048 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:56:59.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=60129542162 2024-09-18T14:56:59.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 60129542162' 2024-09-18T14:56:59.050 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 60129542162 2024-09-18T14:56:59.051 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:56:59.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 60129542161 -lt 60129542162 2024-09-18T14:56:59.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:57:00.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:57:00.359 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:57:00.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 60129542162 -lt 60129542162 2024-09-18T14:57:00.669 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:57:00.670 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:57:00.671 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:57:01.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T14:57:01.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:57:01.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:57:01.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:57:01.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:57:01.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:57:01.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:57:01.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:57:01.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T14:57:01.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:57:01.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:57:01.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:57:01.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T14:57:01.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:57:01.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:57:01.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-09-18T14:57:01.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-09-18T14:57:01.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T14:57:01.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T14:57:01.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T14:57:02.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-18T14:57:02.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T14:57:02.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T14:57:02.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T14:57:02.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T14:57:02.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-18T14:57:02.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T14:57:02.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T14:57:02.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T14:57:02.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T14:57:02.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T14:57:02.150 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:57:02.150 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:57:02.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:57:02.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:57:02.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:57:02.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:57:02.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-18T14:57:02.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T14:57:02.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T14:57:02.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T14:57:02.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T14:57:02.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T14:57:02.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-09-18T14:57:04.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:57:04.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T14:57:04.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:57:04.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T14:57:04.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:57:04.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T14:57:04.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T14:57:04.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:57:04.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:57:04.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:57:04.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T14:57:04.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T14:57:04.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:57:04.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:57:04.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T14:57:04.504 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:57:04.504 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:57:04.504 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:57:04.505 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:57:04.506 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:57:04.506 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:57:04.506 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:57:04.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:57:04.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:57:04.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T14:57:04.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T14:57:04.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:57:04.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:57:04.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:57:04.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:57:04.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:57:04.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T14:57:04.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T14:57:04.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T14:57:04.507 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T14:57:04.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:57:04.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T14:57:04.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T14:57:04.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:57:04.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:57:04.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:57:04.542 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:57:04.533+0000 7f8cfdf1c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:57:04.546 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:57:04.537+0000 7f8cfdf1c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:57:04.548 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:57:04.541+0000 7f8cfdf1c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:57:04.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T14:57:04.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:57:04.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:57:04.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:57:04.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:57:04.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:57:04.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:57:04.825 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T14:57:04.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:57:04.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:57:05.082 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:57:05.073+0000 7f8cfdf1c540 -1 Falling back to public interface 2024-09-18T14:57:05.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:57:06.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:57:06.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:57:06.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:57:06.180 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:57:06.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:57:06.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:57:06.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:57:07.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:57:07.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:57:07.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:57:07.505 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T14:57:07.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:57:07.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:57:07.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:57:07.920 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:57:07.913+0000 7f8cfdf1c540 -1 osd.1 30 log_to_monitors true 2024-09-18T14:57:08.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:57:08.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:57:08.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:57:08.818 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T14:57:08.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:57:08.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:57:09.262 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 34 up_thru 34 down_at 31 last_clean_interval [9,30) [v2:127.0.0.1:6810/165281,v1:127.0.0.1:6811/165281] [v2:127.0.0.1:6812/165281,v1:127.0.0.1:6813/165281] exists,up e505ad50-da22-495a-91f7-4ce2d5176fe2 2024-09-18T14:57:09.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:57:09.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:57:09.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:57:09.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T14:57:09.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:57:09.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:57:09.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:57:09.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T14:57:09.264 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:57:09.264 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:57:09.264 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:57:09.264 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:57:09.264 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:57:09.265 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:57:09.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:57:09.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:57:09.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:57:09.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:57:09.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:57:09.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:57:09.792 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:57:09.792 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T14:57:09.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:57:09.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:57:09.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:57:09.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051589 2024-09-18T14:57:09.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051589 2024-09-18T14:57:09.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051589' 2024-09-18T14:57:09.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:57:09.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:57:10.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=146028888066 2024-09-18T14:57:10.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 146028888066 2024-09-18T14:57:10.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051589 1-146028888066' 2024-09-18T14:57:10.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:57:10.111 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:57:10.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=60129542165 2024-09-18T14:57:10.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 60129542165 2024-09-18T14:57:10.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051589 1-146028888066 2-60129542165' 2024-09-18T14:57:10.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:57:10.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-124554051589 2024-09-18T14:57:10.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:57:10.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:57:10.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-124554051589 2024-09-18T14:57:10.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:57:10.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051589 2024-09-18T14:57:10.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 124554051589' 2024-09-18T14:57:10.256 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 124554051589 2024-09-18T14:57:10.256 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:57:10.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051589 -lt 124554051589 2024-09-18T14:57:10.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:57:10.577 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-146028888066 2024-09-18T14:57:10.577 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:57:10.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:57:10.580 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-146028888066 2024-09-18T14:57:10.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:57:10.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=146028888066 2024-09-18T14:57:10.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 146028888066' 2024-09-18T14:57:10.582 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 146028888066 2024-09-18T14:57:10.583 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:57:10.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 146028888066 -lt 146028888066 2024-09-18T14:57:10.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:57:10.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-60129542165 2024-09-18T14:57:10.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:57:10.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:57:10.927 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-60129542165 2024-09-18T14:57:10.927 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:57:10.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=60129542165 2024-09-18T14:57:10.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 60129542165' 2024-09-18T14:57:10.929 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 60129542165 2024-09-18T14:57:10.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:57:11.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 60129542165 -lt 60129542165 2024-09-18T14:57:11.240 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:57:11.240 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:57:11.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:57:11.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T14:57:11.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:57:11.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:57:11.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:57:11.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:57:11.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:57:11.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:57:11.657 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:57:12.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T14:57:12.007 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:57:12.007 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:57:12.008 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:57:12.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T14:57:12.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:57:12.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:57:12.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: get_pg testpool obj1 2024-09-18T14:57:12.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-09-18T14:57:12.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-09-18T14:57:12.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-09-18T14:57:12.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T14:57:12.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: local pgid=1.0 2024-09-18T14:57:12.723 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: get_primary testpool obj1 2024-09-18T14:57:12.724 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-09-18T14:57:12.724 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T14:57:12.724 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T14:57:12.724 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-09-18T14:57:13.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: local primary=1 2024-09-18T14:57:13.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: get_last_scrub_stamp 1.0 2024-09-18T14:57:13.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T14:57:13.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:57:13.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:57:13.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T14:57:13.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: local last_scrub_stamp=2024-09-18T14:56:27.222987+0000 2024-09-18T14:57:13.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:574: TEST_auto_repair_bluestore_failed: ceph tell 1.0 deep_scrub 2024-09-18T14:57:13.486 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:57:13.487 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": true, 2024-09-18T14:57:13.487 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:55:33.482824+0000" 2024-09-18T14:57:13.487 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:57:13.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:575: TEST_auto_repair_bluestore_failed: ceph tell 1.0 scrub 2024-09-18T14:57:13.625 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T14:57:13.625 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T14:57:13.625 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:55:33.621436+0000" 2024-09-18T14:57:13.625 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T14:57:13.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:578: TEST_auto_repair_bluestore_failed: wait_for_scrub 1.0 2024-09-18T14:56:27.222987+0000 2024-09-18T14:57:13.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-18T14:57:13.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T14:56:27.222987+0000 2024-09-18T14:57:13.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T14:57:13.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T14:57:13.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T14:57:13.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T14:57:13.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T14:57:13.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:57:13.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:57:13.637 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T14:57:13.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:56:27.222987+0000 '>' 2024-09-18T14:56:27.222987+0000 2024-09-18T14:57:13.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T14:57:14.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T14:57:14.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T14:57:14.939 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T14:57:14.939 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T14:57:14.939 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:57:14.939 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:57:14.940 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T14:57:15.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:56:27.222987+0000 '>' 2024-09-18T14:56:27.222987+0000 2024-09-18T14:57:15.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T14:57:16.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T14:57:16.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T14:57:16.256 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T14:57:16.256 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T14:57:16.256 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:57:16.256 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:57:16.257 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T14:57:16.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:56:27.222987+0000 '>' 2024-09-18T14:56:27.222987+0000 2024-09-18T14:57:16.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T14:57:17.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T14:57:17.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T14:57:17.579 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T14:57:17.579 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T14:57:17.579 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:57:17.580 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:57:17.580 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T14:57:17.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:56:27.222987+0000 '>' 2024-09-18T14:56:27.222987+0000 2024-09-18T14:57:17.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T14:57:18.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T14:57:18.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T14:57:18.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T14:57:18.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T14:57:18.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:57:18.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:57:18.899 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T14:57:19.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:56:27.222987+0000 '>' 2024-09-18T14:56:27.222987+0000 2024-09-18T14:57:19.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T14:57:20.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T14:57:20.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T14:57:20.234 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T14:57:20.234 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T14:57:20.235 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:57:20.235 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:57:20.235 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T14:57:20.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:57:15.339206+0000 '>' 2024-09-18T14:56:27.222987+0000 2024-09-18T14:57:20.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T14:57:20.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:579: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-09-18T14:57:20.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:57:20.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:57:20.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:57:20.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T14:57:20.547 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:57:20.547 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:57:20.547 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:57:20.548 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:57:20.548 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:57:20.548 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:57:20.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:57:20.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:57:20.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:57:20.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:57:20.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:57:21.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:57:21.102 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:57:21.102 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T14:57:21.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:57:21.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:57:21.103 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:57:21.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051593 2024-09-18T14:57:21.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051593 2024-09-18T14:57:21.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051593' 2024-09-18T14:57:21.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:57:21.240 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:57:21.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=146028888069 2024-09-18T14:57:21.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 146028888069 2024-09-18T14:57:21.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051593 1-146028888069' 2024-09-18T14:57:21.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:57:21.384 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:57:21.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=60129542168 2024-09-18T14:57:21.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 60129542168 2024-09-18T14:57:21.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051593 1-146028888069 2-60129542168' 2024-09-18T14:57:21.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:57:21.516 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-124554051593 2024-09-18T14:57:21.517 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:57:21.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:57:21.520 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-124554051593 2024-09-18T14:57:21.520 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:57:21.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051593 2024-09-18T14:57:21.523 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 124554051593 2024-09-18T14:57:21.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 124554051593' 2024-09-18T14:57:21.523 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:57:21.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051591 -lt 124554051593 2024-09-18T14:57:21.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:57:22.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:57:22.836 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:57:23.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051593 -lt 124554051593 2024-09-18T14:57:23.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:57:23.157 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-146028888069 2024-09-18T14:57:23.157 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:57:23.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:57:23.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-146028888069 2024-09-18T14:57:23.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:57:23.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=146028888069 2024-09-18T14:57:23.162 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 146028888069 2024-09-18T14:57:23.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 146028888069' 2024-09-18T14:57:23.163 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:57:23.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 146028888069 -lt 146028888069 2024-09-18T14:57:23.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:57:23.510 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-60129542168 2024-09-18T14:57:23.511 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:57:23.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:57:23.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-60129542168 2024-09-18T14:57:23.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:57:23.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=60129542168 2024-09-18T14:57:23.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 60129542168' 2024-09-18T14:57:23.517 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 60129542168 2024-09-18T14:57:23.517 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:57:23.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 60129542168 -lt 60129542168 2024-09-18T14:57:23.856 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:57:23.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:57:23.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:57:24.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T14:57:24.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:57:24.272 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:57:24.272 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:57:24.272 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:57:24.272 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:57:24.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:57:24.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:57:24.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T14:57:24.589 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:57:24.589 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:57:24.590 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:57:24.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T14:57:24.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:57:24.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:57:24.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:580: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-09-18T14:57:24.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:57:24.947 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:57:25.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:57:25.303 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:57:25.303 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T14:57:25.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:57:25.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:57:25.303 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:57:25.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051594 2024-09-18T14:57:25.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051594 2024-09-18T14:57:25.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051594' 2024-09-18T14:57:25.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:57:25.479 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:57:25.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=146028888071 2024-09-18T14:57:25.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 146028888071 2024-09-18T14:57:25.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051594 1-146028888071' 2024-09-18T14:57:25.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:57:25.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:57:25.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=60129542170 2024-09-18T14:57:25.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 60129542170 2024-09-18T14:57:25.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051594 1-146028888071 2-60129542170' 2024-09-18T14:57:25.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:57:25.774 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-124554051594 2024-09-18T14:57:25.774 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:57:25.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:57:25.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-124554051594 2024-09-18T14:57:25.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:57:25.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051594 2024-09-18T14:57:25.780 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 124554051594 2024-09-18T14:57:25.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 124554051594' 2024-09-18T14:57:25.780 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:57:26.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051593 -lt 124554051594 2024-09-18T14:57:26.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:57:27.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:57:27.092 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:57:27.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051595 -lt 124554051594 2024-09-18T14:57:27.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:57:27.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-146028888071 2024-09-18T14:57:27.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:57:27.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:57:27.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-146028888071 2024-09-18T14:57:27.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:57:27.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=146028888071 2024-09-18T14:57:27.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 146028888071' 2024-09-18T14:57:27.413 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 146028888071 2024-09-18T14:57:27.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:57:27.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 146028888071 -lt 146028888071 2024-09-18T14:57:27.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:57:27.755 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-60129542170 2024-09-18T14:57:27.755 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:57:27.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:57:27.758 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-60129542170 2024-09-18T14:57:27.758 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:57:27.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=60129542170 2024-09-18T14:57:27.760 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 60129542170 2024-09-18T14:57:27.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 60129542170' 2024-09-18T14:57:27.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:57:28.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 60129542170 -lt 60129542170 2024-09-18T14:57:28.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:581: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-18T14:57:28.113 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:57:15.117+0000 7f8ce1c2f700 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber : scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-18T14:57:28.113 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:57:15.117+0000 7f8ce1c2f700 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber : _scrub_finish info stats: valid 2024-09-18T14:57:28.113 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:57:15.117+0000 7f8ce1c2f700 20 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber : scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-09-18T14:57:28.113 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:57:15.333+0000 7f8ce1c2f700 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber : scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-09-18T14:57:28.113 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:57:15.333+0000 7f8ce1c2f700 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber : _scrub_finish info stats: valid 2024-09-18T14:57:28.114 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:57:15.333+0000 7f8ce1c2f700 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber : scrub_finish 3 error(s) still present after re-scrub 2024-09-18T14:57:28.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:582: TEST_auto_repair_bluestore_failed: grep -q 'scrub_finish.*still present after re-scrub' td/osd-scrub-repair/osd.1.log 2024-09-18T14:57:28.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-09-18T14:57:28.444 INFO:tasks.workunit.client.0.smithi046.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:57:28.444 INFO:tasks.workunit.client.0.smithi046.stdout:1.0 10 0 0 0 0 63 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-09-18T14:57:15.339378+0000 20'10 35:115 [1,0] 1 [1,0] 1 20'10 2024-09-18T14:57:15.339206+0000 20'10 2024-09-18T14:57:15.339206+0000 0 1 periodic scrub scheduled @ 2024-09-19T14:57:15.339206+0000 10 0 2024-09-18T14:57:28.444 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-09-18T14:57:28.444 INFO:tasks.workunit.client.0.smithi046.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:57:28.447 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:57:28.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:584: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-09-18T14:57:28.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:584: TEST_auto_repair_bluestore_failed: grep -q '^1.0.*+failed_repair' 2024-09-18T14:57:28.745 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:57:28.755 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:588: TEST_auto_repair_bluestore_failed: get_not_primary testpool obj1 2024-09-18T14:57:28.755 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-09-18T14:57:28.755 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-09-18T14:57:28.756 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-09-18T14:57:28.756 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-09-18T14:57:28.756 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T14:57:28.756 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T14:57:28.756 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-09-18T14:57:29.117 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T14:57:29.117 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-09-18T14:57:29.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T14:57:29.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:588: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 list-attrs 2024-09-18T14:57:29.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T14:57:29.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T14:57:29.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T14:57:29.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T14:57:29.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 list-attrs 2024-09-18T14:57:29.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T14:57:29.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T14:57:29.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T14:57:29.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T14:57:29.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T14:57:29.515 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:57:29.515 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:57:29.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:57:29.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:57:29.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:57:29.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:57:29.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 list-attrs 2024-09-18T14:57:29.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T14:57:29.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T14:57:29.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T14:57:29.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T14:57:29.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T14:57:29.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 list-attrs 2024-09-18T14:57:31.455 INFO:tasks.workunit.client.0.smithi046.stdout:_ 2024-09-18T14:57:31.455 INFO:tasks.workunit.client.0.smithi046.stdout:snapset 2024-09-18T14:57:31.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:57:31.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T14:57:31.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:57:31.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T14:57:31.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:57:31.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T14:57:31.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T14:57:31.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:57:31.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:57:31.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:57:31.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T14:57:31.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T14:57:31.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:57:31.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:57:31.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T14:57:31.992 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:57:31.992 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:57:31.992 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:57:31.994 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:57:31.994 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:57:31.994 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:57:31.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:57:31.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:57:31.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:57:31.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T14:57:31.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T14:57:31.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:57:31.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:57:31.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:57:31.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:57:31.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:57:31.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T14:57:31.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T14:57:31.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T14:57:31.996 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T14:57:31.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:57:31.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T14:57:31.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T14:57:31.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:57:31.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:57:31.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:57:32.030 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:57:32.021+0000 7f309bbb8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:57:32.033 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:57:32.025+0000 7f309bbb8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:57:32.035 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:57:32.025+0000 7f309bbb8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:57:32.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T14:57:32.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:57:32.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:57:32.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:57:32.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:57:32.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:57:32.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:57:32.310 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T14:57:32.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:57:32.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:57:32.570 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:57:32.560+0000 7f309bbb8540 -1 Falling back to public interface 2024-09-18T14:57:32.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:57:33.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:57:33.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:57:33.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:57:33.677 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:57:33.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:57:33.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:57:33.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:57:34.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:57:34.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:57:34.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:57:34.994 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T14:57:34.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:57:34.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:57:35.303 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:57:35.292+0000 7f309bbb8540 -1 osd.0 35 log_to_monitors true 2024-09-18T14:57:35.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:57:36.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:57:36.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:57:36.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:57:36.449 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T14:57:36.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:57:36.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:57:36.801 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 39 up_thru 31 down_at 36 last_clean_interval [29,35) [v2:127.0.0.1:6802/167571,v1:127.0.0.1:6803/167571] [v2:127.0.0.1:6804/167571,v1:127.0.0.1:6805/167571] exists,up 84578cdf-ea8c-423e-916f-6ca29549de31 2024-09-18T14:57:36.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:57:36.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:57:36.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:57:36.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T14:57:36.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:57:36.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:57:36.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:57:36.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T14:57:36.804 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:57:36.804 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:57:36.804 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:57:36.804 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:57:36.804 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:57:36.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:57:37.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:57:37.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:57:37.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:57:37.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:57:37.024 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:57:37.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:57:37.340 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:57:37.340 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T14:57:37.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:57:37.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:57:37.341 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:57:37.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=167503724546 2024-09-18T14:57:37.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 167503724546 2024-09-18T14:57:37.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-167503724546' 2024-09-18T14:57:37.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:57:37.553 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:57:37.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=146028888074 2024-09-18T14:57:37.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 146028888074 2024-09-18T14:57:37.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-167503724546 1-146028888074' 2024-09-18T14:57:37.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:57:37.690 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:57:37.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=60129542174 2024-09-18T14:57:37.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 60129542174 2024-09-18T14:57:37.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-167503724546 1-146028888074 2-60129542174' 2024-09-18T14:57:37.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:57:37.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-167503724546 2024-09-18T14:57:37.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:57:37.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:57:37.837 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-167503724546 2024-09-18T14:57:37.837 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:57:37.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=167503724546 2024-09-18T14:57:37.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 167503724546' 2024-09-18T14:57:37.838 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 167503724546 2024-09-18T14:57:37.839 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:57:38.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 167503724546 2024-09-18T14:57:38.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:57:39.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:57:39.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:57:39.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 167503724546 -lt 167503724546 2024-09-18T14:57:39.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:57:39.468 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-146028888074 2024-09-18T14:57:39.469 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:57:39.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:57:39.471 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-146028888074 2024-09-18T14:57:39.471 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:57:39.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=146028888074 2024-09-18T14:57:39.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 146028888074' 2024-09-18T14:57:39.473 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 146028888074 2024-09-18T14:57:39.473 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:57:39.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 146028888074 -lt 146028888074 2024-09-18T14:57:39.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:57:39.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-60129542174 2024-09-18T14:57:39.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:57:39.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:57:39.820 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-60129542174 2024-09-18T14:57:39.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:57:39.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=60129542174 2024-09-18T14:57:39.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 60129542174' 2024-09-18T14:57:39.822 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 60129542174 2024-09-18T14:57:39.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:57:40.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 60129542174 -lt 60129542174 2024-09-18T14:57:40.168 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:57:40.168 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:57:40.169 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:57:40.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T14:57:40.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:57:40.586 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:57:40.586 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:57:40.586 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:57:40.586 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:57:40.586 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:57:40.587 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:57:40.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T14:57:40.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:57:40.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:57:40.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:57:41.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T14:57:41.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:57:41.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:57:41.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:589: TEST_auto_repair_bluestore_failed: rados --pool testpool get obj1 td/osd-scrub-repair/COPY 2024-09-18T14:57:41.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:590: TEST_auto_repair_bluestore_failed: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-18T14:57:41.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:591: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-18T14:57:41.339 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:57:15.117+0000 7f8ce1c2f700 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber : scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-18T14:57:41.339 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:57:15.117+0000 7f8ce1c2f700 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber : _scrub_finish info stats: valid 2024-09-18T14:57:41.339 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:57:15.117+0000 7f8ce1c2f700 20 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber : scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-09-18T14:57:41.339 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:57:15.333+0000 7f8ce1c2f700 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber : scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-09-18T14:57:41.339 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:57:15.333+0000 7f8ce1c2f700 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber : _scrub_finish info stats: valid 2024-09-18T14:57:41.339 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:57:15.333+0000 7f8ce1c2f700 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber : scrub_finish 3 error(s) still present after re-scrub 2024-09-18T14:57:41.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:594: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-09-18T14:57:41.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-09-18T14:57:41.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T14:57:41.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T14:57:41.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T14:57:41.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:594: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 remove 2024-09-18T14:57:41.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T14:57:41.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T14:57:41.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T14:57:41.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T14:57:41.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 remove 2024-09-18T14:57:41.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T14:57:41.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T14:57:41.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T14:57:41.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T14:57:41.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T14:57:41.640 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:57:41.640 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:57:41.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:57:41.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:57:41.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:57:41.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:57:41.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 remove 2024-09-18T14:57:41.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T14:57:41.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T14:57:41.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T14:57:41.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T14:57:41.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T14:57:41.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 remove 2024-09-18T14:57:43.454 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:104778fc:::obj2:head#, (61) No data available 2024-09-18T14:57:43.455 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:104778fc:::obj2:head# 2024-09-18T14:57:43.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:57:43.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T14:57:43.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:57:43.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T14:57:43.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:57:43.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T14:57:43.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T14:57:43.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:57:43.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:57:43.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:57:43.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T14:57:43.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T14:57:43.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:57:43.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:57:43.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T14:57:44.000 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:57:44.000 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:57:44.000 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:57:44.001 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:57:44.001 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:57:44.001 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:57:44.001 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:57:44.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:57:44.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:57:44.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T14:57:44.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T14:57:44.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:57:44.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:57:44.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:57:44.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:57:44.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:57:44.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T14:57:44.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T14:57:44.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T14:57:44.003 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T14:57:44.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:57:44.004 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T14:57:44.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T14:57:44.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:57:44.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:57:44.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:57:44.036 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:57:44.028+0000 7f347bcfd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:57:44.039 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:57:44.032+0000 7f347bcfd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:57:44.041 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:57:44.032+0000 7f347bcfd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:57:44.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T14:57:44.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:57:44.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:57:44.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:57:44.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:57:44.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:57:44.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:57:44.319 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T14:57:44.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:57:44.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:57:44.582 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:57:44.572+0000 7f347bcfd540 -1 Falling back to public interface 2024-09-18T14:57:44.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:57:45.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:57:45.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:57:45.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:57:45.674 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:57:45.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:57:45.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:57:45.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:57:46.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:57:46.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:57:46.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:57:46.992 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T14:57:46.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:57:46.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:57:47.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:57:47.468 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:57:47.460+0000 7f347bcfd540 -1 osd.1 40 log_to_monitors true 2024-09-18T14:57:48.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:57:48.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:57:48.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:57:48.315 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T14:57:48.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:57:48.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:57:48.637 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 41 last_clean_interval [34,40) [v2:127.0.0.1:6810/168972,v1:127.0.0.1:6811/168972] [v2:127.0.0.1:6812/168972,v1:127.0.0.1:6813/168972] exists,up e505ad50-da22-495a-91f7-4ce2d5176fe2 2024-09-18T14:57:48.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:57:48.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:57:48.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:57:48.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T14:57:48.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:57:48.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:57:48.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:57:48.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T14:57:48.639 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:57:48.639 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:57:48.639 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:57:48.640 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:57:48.640 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:57:48.640 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:57:48.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:57:48.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:57:48.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:57:48.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:57:48.859 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:57:49.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:57:49.170 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:57:49.170 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T14:57:49.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:57:49.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:57:49.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:57:49.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=167503724549 2024-09-18T14:57:49.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 167503724549 2024-09-18T14:57:49.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-167503724549' 2024-09-18T14:57:49.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:57:49.364 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:57:49.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=188978561026 2024-09-18T14:57:49.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 188978561026 2024-09-18T14:57:49.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-167503724549 1-188978561026' 2024-09-18T14:57:49.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:57:49.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:57:49.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=60129542177 2024-09-18T14:57:49.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 60129542177 2024-09-18T14:57:49.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-167503724549 1-188978561026 2-60129542177' 2024-09-18T14:57:49.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:57:49.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-167503724549 2024-09-18T14:57:49.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:57:49.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:57:49.659 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-167503724549 2024-09-18T14:57:49.659 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:57:49.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=167503724549 2024-09-18T14:57:49.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 167503724549' 2024-09-18T14:57:49.660 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 167503724549 2024-09-18T14:57:49.661 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:57:49.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 167503724548 -lt 167503724549 2024-09-18T14:57:49.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:57:50.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:57:50.969 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:57:51.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 167503724549 -lt 167503724549 2024-09-18T14:57:51.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:57:51.334 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-188978561026 2024-09-18T14:57:51.335 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:57:51.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:57:51.338 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-188978561026 2024-09-18T14:57:51.338 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:57:51.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=188978561026 2024-09-18T14:57:51.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 188978561026' 2024-09-18T14:57:51.339 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 188978561026 2024-09-18T14:57:51.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:57:51.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 188978561026 -lt 188978561026 2024-09-18T14:57:51.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:57:51.682 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-60129542177 2024-09-18T14:57:51.683 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:57:51.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:57:51.686 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-60129542177 2024-09-18T14:57:51.686 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:57:51.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=60129542177 2024-09-18T14:57:51.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 60129542177' 2024-09-18T14:57:51.688 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 60129542177 2024-09-18T14:57:51.689 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:57:52.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 60129542177 -lt 60129542177 2024-09-18T14:57:52.040 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:57:52.041 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:57:52.041 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:57:52.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T14:57:52.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:57:52.454 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:57:52.454 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:57:52.454 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:57:52.454 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:57:52.455 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:57:52.455 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:57:52.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T14:57:52.762 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:57:52.762 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:57:52.763 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:57:53.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T14:57:53.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:57:53.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:57:53.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:595: TEST_auto_repair_bluestore_failed: repair 1.0 2024-09-18T14:57:53.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=1.0 2024-09-18T14:57:53.142 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 1.0 2024-09-18T14:57:53.142 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T14:57:53.142 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:57:53.142 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:57:53.142 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T14:57:53.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T14:57:15.339206+0000 2024-09-18T14:57:53.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 1.0 2024-09-18T14:57:53.762 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 1.0 on osd.1 to repair 2024-09-18T14:57:53.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 1.0 2024-09-18T14:57:15.339206+0000 2024-09-18T14:57:53.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-18T14:57:53.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T14:57:15.339206+0000 2024-09-18T14:57:53.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T14:57:53.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T14:57:53.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T14:57:53.772 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T14:57:53.772 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T14:57:53.772 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:57:53.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:57:53.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T14:57:54.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:57:15.339206+0000 '>' 2024-09-18T14:57:15.339206+0000 2024-09-18T14:57:54.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T14:57:55.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T14:57:55.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T14:57:55.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T14:57:55.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T14:57:55.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:57:55.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:57:55.087 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T14:57:55.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:57:15.339206+0000 '>' 2024-09-18T14:57:15.339206+0000 2024-09-18T14:57:55.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T14:57:56.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T14:57:56.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T14:57:56.402 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T14:57:56.402 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T14:57:56.402 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:57:56.403 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:57:56.403 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T14:57:56.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:57:15.339206+0000 '>' 2024-09-18T14:57:15.339206+0000 2024-09-18T14:57:56.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T14:57:57.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T14:57:57.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T14:57:57.744 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T14:57:57.744 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T14:57:57.744 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:57:57.744 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:57:57.744 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T14:57:58.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:57:15.339206+0000 '>' 2024-09-18T14:57:15.339206+0000 2024-09-18T14:57:58.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T14:57:59.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T14:57:59.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T14:57:59.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T14:57:59.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T14:57:59.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T14:57:59.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T14:57:59.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T14:57:59.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:57:56.471609+0000 '>' 2024-09-18T14:57:15.339206+0000 2024-09-18T14:57:59.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T14:57:59.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:596: TEST_auto_repair_bluestore_failed: sleep 2 2024-09-18T14:58:01.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:598: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-09-18T14:58:01.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:58:01.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:58:01.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:58:01.699 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:58:01.699 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T14:58:01.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:58:01.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:58:01.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:58:01.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=167503724553 2024-09-18T14:58:01.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 167503724553 2024-09-18T14:58:01.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-167503724553' 2024-09-18T14:58:01.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:58:01.922 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:58:02.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=188978561029 2024-09-18T14:58:02.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 188978561029 2024-09-18T14:58:02.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-167503724553 1-188978561029' 2024-09-18T14:58:02.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:58:02.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:58:02.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=60129542180 2024-09-18T14:58:02.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 60129542180 2024-09-18T14:58:02.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-167503724553 1-188978561029 2-60129542180' 2024-09-18T14:58:02.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:58:02.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-167503724553 2024-09-18T14:58:02.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:58:02.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:58:02.196 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-167503724553 2024-09-18T14:58:02.196 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:58:02.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=167503724553 2024-09-18T14:58:02.199 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 167503724553 2024-09-18T14:58:02.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 167503724553' 2024-09-18T14:58:02.199 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:58:02.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 167503724553 -lt 167503724553 2024-09-18T14:58:02.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:58:02.510 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-188978561029 2024-09-18T14:58:02.510 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:58:02.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:58:02.513 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-188978561029 2024-09-18T14:58:02.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:58:02.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=188978561029 2024-09-18T14:58:02.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 188978561029' 2024-09-18T14:58:02.516 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 188978561029 2024-09-18T14:58:02.516 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:58:02.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 188978561029 -lt 188978561029 2024-09-18T14:58:02.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:58:02.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-60129542180 2024-09-18T14:58:02.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:58:02.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:58:02.866 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-60129542180 2024-09-18T14:58:02.866 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:58:02.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=60129542180 2024-09-18T14:58:02.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 60129542180' 2024-09-18T14:58:02.868 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 60129542180 2024-09-18T14:58:02.868 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:58:03.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 60129542180 -lt 60129542180 2024-09-18T14:58:03.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-09-18T14:58:03.514 INFO:tasks.workunit.client.0.smithi046.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:58:03.514 INFO:tasks.workunit.client.0.smithi046.stdout:1.0 9 0 0 0 0 63 0 0 10 0 10 active+clean 2024-09-18T14:57:56.471770+0000 20'10 45:159 [1,0] 1 [1,0] 1 20'10 2024-09-18T14:57:56.471609+0000 20'10 2024-09-18T14:57:56.471609+0000 0 1 periodic scrub scheduled @ 2024-09-19T14:57:56.471609+0000 9 0 2024-09-18T14:58:03.514 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-09-18T14:58:03.514 INFO:tasks.workunit.client.0.smithi046.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:58:03.520 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:58:03.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:600: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-09-18T14:58:03.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:600: TEST_auto_repair_bluestore_failed: grep -q -e '^1.0.* active+clean ' -e '^1.0.* active+clean+wait ' 2024-09-18T14:58:03.822 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T14:58:03.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:601: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-18T14:58:03.836 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:57:15.117+0000 7f8ce1c2f700 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber : scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-18T14:58:03.836 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:57:15.117+0000 7f8ce1c2f700 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber : _scrub_finish info stats: valid 2024-09-18T14:58:03.836 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:57:15.117+0000 7f8ce1c2f700 20 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber : scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-09-18T14:58:03.836 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:57:15.333+0000 7f8ce1c2f700 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber : scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-09-18T14:58:03.836 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:57:15.333+0000 7f8ce1c2f700 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber : _scrub_finish info stats: valid 2024-09-18T14:58:03.836 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:57:15.333+0000 7f8ce1c2f700 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=15/15 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber : scrub_finish 3 error(s) still present after re-scrub 2024-09-18T14:58:03.836 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:57:56.464+0000 7f345fa10700 10 osd.1 pg_epoch: 45 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=44/45 n=10 ec=15/15 lis/c=44/44 les/c/f=45/45/0 sis=44) [1,0] r=0 lpr=44 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: REQ_SCRUB ] MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB] scrubber : scrub_finish before flags: REQ_SCRUB. repair state: repair. deep_scrub_on_error: 0 2024-09-18T14:58:03.837 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:57:56.464+0000 7f345fa10700 10 osd.1 pg_epoch: 45 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=44/45 n=10 ec=15/15 lis/c=44/44 les/c/f=45/45/0 sis=44) [1,0] r=0 lpr=44 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: REQ_SCRUB ] ] scrubber : _scrub_finish info stats: valid 2024-09-18T14:58:03.837 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T14:57:56.464+0000 7f345fa10700 20 osd.1 pg_epoch: 45 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=44/45 n=9 ec=15/15 lis/c=44/44 les/c/f=45/45/0 sis=44) [1,0] r=0 lpr=44 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: REQ_SCRUB ] ] scrubber : scrub_finish All may be fixed 2024-09-18T14:58:03.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T14:58:03.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T14:58:03.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:58:03.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T14:58:03.838 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:58:03.838 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:58:03.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:58:03.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:58:03.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:58:03.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:58:03.958 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:58:03.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:58:03.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:58:03.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:58:03.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:58:03.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:58:03.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:58:03.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:58:03.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:58:03.967 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:58:03.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:58:03.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:58:03.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T14:58:03.995 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:58:03.995 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:58:03.995 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:58:03.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T14:58:03.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:58:03.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:58:03.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T14:58:03.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T14:58:03.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T14:58:03.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T14:58:03.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T14:58:03.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:58:03.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T14:58:03.999 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:58:03.999 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:58:04.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:58:04.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:58:04.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:58:04.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:58:04.004 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:58:04.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:58:04.006 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:58:04.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:58:04.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:58:04.008 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:58:04.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:58:04.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:58:04.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:58:04.012 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:58:04.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:58:04.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:58:04.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T14:58:04.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:58:04.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:58:04.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:58:04.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T14:58:04.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:58:04.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:58:04.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T14:58:04.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:58:04.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:58:04.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:58:04.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T14:58:04.025 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:58:04.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T14:58:04.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:58:04.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T14:58:04.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_auto_repair_bluestore_failed_norecov td/osd-scrub-repair 2024-09-18T14:58:04.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:605: TEST_auto_repair_bluestore_failed_norecov: local dir=td/osd-scrub-repair 2024-09-18T14:58:04.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:606: TEST_auto_repair_bluestore_failed_norecov: local poolname=testpool 2024-09-18T14:58:04.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:609: TEST_auto_repair_bluestore_failed_norecov: run_mon td/osd-scrub-repair a 2024-09-18T14:58:04.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T14:58:04.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:58:04.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:58:04.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:58:04.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T14:58:04.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-18T14:58:04.430 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:58:04.430 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:58:04.430 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:58:04.431 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:58:04.431 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:58:04.431 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:58:04.431 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:58:04.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T14:58:04.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:58:04.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:58:04.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:58:04.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:58:04.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:58:04.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:58:04.749 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:58:04.750 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:58:04.750 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:58:04.750 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:58:04.750 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:58:04.750 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:58:04.750 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T14:58:04.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:58:04.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T14:58:04.845 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:58:04.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:58:04.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:58:04.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:58:04.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:58:04.846 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:58:04.846 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:58:04.847 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:58:04.847 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:58:04.847 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:58:04.847 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:58:04.848 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T14:58:04.848 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:58:04.848 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T14:58:04.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:610: TEST_auto_repair_bluestore_failed_norecov: run_mgr td/osd-scrub-repair x 2024-09-18T14:58:04.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T14:58:04.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:58:04.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:58:04.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:58:04.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T14:58:04.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:58:05.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:58:05.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:58:05.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:58:05.813 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:58:05.813 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:58:05.813 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:58:05.814 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:58:05.815 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:58:05.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:58:05.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T14:58:05.853 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: seq 0 2 2024-09-18T14:58:05.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-09-18T14:58:05.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:615: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:58:05.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T14:58:05.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:58:05.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:58:05.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:58:05.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T14:58:05.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T14:58:05.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:58:05.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:58:05.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:58:05.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T14:58:05.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T14:58:05.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:58:05.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:58:05.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T14:58:05.859 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:58:05.859 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:58:05.859 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:58:05.859 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:58:05.859 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:58:05.860 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:58:05.860 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:58:05.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:58:05.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:58:05.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:58:05.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:58:05.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T14:58:05.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T14:58:05.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:58:05.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:58:05.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:58:05.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:58:05.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:58:05.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T14:58:05.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T14:58:05.862 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:58:05.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=33c23184-ae55-4b17-9812-4bddeeecfc1d 2024-09-18T14:58:05.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 33c23184-ae55-4b17-9812-4bddeeecfc1d' 2024-09-18T14:58:05.864 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 33c23184-ae55-4b17-9812-4bddeeecfc1d 2024-09-18T14:58:05.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:58:05.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB96upmQdR9NBAAX4xmg93kiZF4r2IC4gI5hg== 2024-09-18T14:58:05.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB96upmQdR9NBAAX4xmg93kiZF4r2IC4gI5hg=="}' 2024-09-18T14:58:05.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 33c23184-ae55-4b17-9812-4bddeeecfc1d -i td/osd-scrub-repair/0/new.json 2024-09-18T14:58:06.155 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:58:06.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-18T14:58:06.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQB96upmQdR9NBAAX4xmg93kiZF4r2IC4gI5hg== --osd-uuid 33c23184-ae55-4b17-9812-4bddeeecfc1d 2024-09-18T14:58:06.203 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:58:06.192+0000 7f4754ce4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:58:06.206 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:58:06.196+0000 7f4754ce4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:58:06.208 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:58:06.200+0000 7f4754ce4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:58:06.209 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:58:06.200+0000 7f4754ce4540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T14:58:06.209 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:58:06.200+0000 7f4754ce4540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T14:58:06.209 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:58:06.200+0000 7f4754ce4540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T14:58:06.278 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:58:06.268+0000 7f4754ce4540 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-18T14:58:12.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T14:58:12.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:58:12.923 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T14:58:12.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:58:12.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:58:13.345 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T14:58:13.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:58:13.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:58:13.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:58:13.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:58:13.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:58:13.373 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:58:13.364+0000 7f1715e62540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:58:13.399 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:58:13.388+0000 7f1715e62540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:58:13.401 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:58:13.392+0000 7f1715e62540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:58:13.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:58:13.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:58:13.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:58:13.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:58:13.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:58:13.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:13.648 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:58:13.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:58:13.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:13.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:58:13.945 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:58:13.936+0000 7f1715e62540 -1 Falling back to public interface 2024-09-18T14:58:13.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:58:14.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:58:14.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:14.995 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:58:14.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:58:14.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:14.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:58:15.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:58:16.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:58:16.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:16.305 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:58:16.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:58:16.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:16.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:58:16.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:58:16.690 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:58:16.680+0000 7f1715e62540 -1 osd.0 0 log_to_monitors true 2024-09-18T14:58:17.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:58:17.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:17.624 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:58:17.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:58:17.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:17.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:58:17.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:58:18.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:58:18.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:18.944 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:58:18.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:58:18.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:18.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:58:19.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:58:20.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:58:20.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:20.342 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:58:20.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:58:20.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:20.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:58:20.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:58:21.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:58:21.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:21.710 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:58:21.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:58:21.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:21.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:58:22.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:58:23.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:58:23.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:23.029 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:58:23.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:58:23.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:23.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:58:23.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:58:24.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:58:24.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:24.354 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T14:58:24.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:58:24.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:24.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:58:24.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:58:25.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:58:25.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:25.793 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T14:58:25.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:58:25.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:25.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:58:26.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:58:27.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:58:27.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:27.111 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T14:58:27.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:58:27.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:27.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:58:27.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:58:28.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:58:28.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:28.537 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T14:58:28.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:58:28.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:28.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:58:28.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:58:29.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:58:29.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:29.858 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T14:58:29.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:58:29.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:29.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:58:30.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:58:31.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:58:31.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:31.302 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T14:58:31.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:58:31.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:31.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:58:31.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:58:32.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:58:32.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:32.727 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T14:58:32.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:58:32.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:32.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:58:33.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:58:34.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:58:34.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:34.158 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T14:58:34.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:58:34.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:34.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:58:34.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:58:35.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:58:35.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:35.973 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T14:58:35.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:58:35.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:35.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:58:36.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:58:37.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:58:37.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:37.389 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T14:58:37.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:58:37.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:37.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:58:38.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:58:39.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:58:39.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:39.113 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T14:58:39.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:58:39.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:39.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:58:39.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:58:40.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:58:40.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:40.957 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T14:58:40.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T14:58:40.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:40.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:58:41.277 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/171468,v1:127.0.0.1:6803/171468] [v2:127.0.0.1:6804/171468,v1:127.0.0.1:6805/171468] exists,up 33c23184-ae55-4b17-9812-4bddeeecfc1d 2024-09-18T14:58:41.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:58:41.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:58:41.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:58:41.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-09-18T14:58:41.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:615: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:58:41.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T14:58:41.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:58:41.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T14:58:41.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:58:41.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T14:58:41.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T14:58:41.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:58:41.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:58:41.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:58:41.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T14:58:41.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T14:58:41.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:58:41.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:58:41.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T14:58:41.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:58:41.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:58:41.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:58:41.281 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:58:41.282 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:58:41.282 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:58:41.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:58:41.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:58:41.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:58:41.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:58:41.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:58:41.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T14:58:41.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T14:58:41.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:58:41.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:58:41.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:58:41.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:58:41.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:58:41.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T14:58:41.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T14:58:41.285 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:58:41.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e3412a06-2285-4d0b-af4a-8086931e9f3d 2024-09-18T14:58:41.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e3412a06-2285-4d0b-af4a-8086931e9f3d' 2024-09-18T14:58:41.286 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 e3412a06-2285-4d0b-af4a-8086931e9f3d 2024-09-18T14:58:41.286 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:58:41.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCh6upmKjrkERAAHcu4+isPoV/VdQMQ/FMY5A== 2024-09-18T14:58:41.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCh6upmKjrkERAAHcu4+isPoV/VdQMQ/FMY5A=="}' 2024-09-18T14:58:41.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e3412a06-2285-4d0b-af4a-8086931e9f3d -i td/osd-scrub-repair/1/new.json 2024-09-18T14:58:41.732 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:58:41.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-18T14:58:41.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCh6upmKjrkERAAHcu4+isPoV/VdQMQ/FMY5A== --osd-uuid e3412a06-2285-4d0b-af4a-8086931e9f3d 2024-09-18T14:58:41.785 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:58:41.780+0000 7f3e7d0b0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:58:41.790 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:58:41.784+0000 7f3e7d0b0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:58:41.792 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:58:41.788+0000 7f3e7d0b0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:58:41.793 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:58:41.788+0000 7f3e7d0b0540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T14:58:41.793 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:58:41.788+0000 7f3e7d0b0540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T14:58:41.793 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:58:41.788+0000 7f3e7d0b0540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T14:58:41.901 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:58:41.896+0000 7f3e7d0b0540 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-18T14:58:48.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-18T14:58:48.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:58:48.157 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T14:58:48.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T14:58:48.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:58:48.546 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T14:58:48.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T14:58:48.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:58:48.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:58:48.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:58:48.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:58:48.580 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:58:48.575+0000 7f23e07ce540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:58:48.582 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:58:48.575+0000 7f23e07ce540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:58:48.583 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:58:48.575+0000 7f23e07ce540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:58:48.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T14:58:48.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:58:48.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:58:48.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:58:48.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:58:48.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:48.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:58:48.875 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:58:48.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:48.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:58:49.134 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:58:49.127+0000 7f23e07ce540 -1 Falling back to public interface 2024-09-18T14:58:49.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:58:50.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:58:50.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:50.482 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:58:50.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:58:50.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:50.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:58:50.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:58:51.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:58:51.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:51.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:58:51.814 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:58:51.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:51.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:58:51.963 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:58:51.947+0000 7f23e07ce540 -1 osd.1 0 log_to_monitors true 2024-09-18T14:58:52.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:58:53.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:58:53.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:53.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:58:53.177 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:58:53.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:53.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:58:53.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:58:54.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:58:54.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:54.491 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:58:54.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:58:54.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:54.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:58:54.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:58:55.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:58:55.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:55.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:58:55.944 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:58:55.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:55.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:58:56.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:58:57.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:58:57.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:57.384 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:58:57.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:58:57.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:58:57.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:57.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:58:58.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:58:58.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:58:58.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:58:58.708 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:58:58.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:58:58.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:58:59.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:00.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:00.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:00.270 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T14:59:00.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:59:00.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:00.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:59:00.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:01.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:01.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:01.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:59:01.400 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T14:59:01.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:01.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:59:01.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:02.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:02.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:02.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:59:02.715 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T14:59:02.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:02.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:59:03.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:04.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:04.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:04.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:59:04.079 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T14:59:04.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:04.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:59:04.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:05.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:05.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:05.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:59:05.420 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T14:59:05.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:05.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:59:05.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:06.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:06.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:06.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:59:06.892 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T14:59:06.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:06.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:59:07.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:08.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:08.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:08.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:59:08.568 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T14:59:08.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:08.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:59:09.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:10.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:10.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:10.022 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T14:59:10.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:59:10.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:10.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:59:10.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:11.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:11.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:11.656 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T14:59:11.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:59:11.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:11.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:59:12.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:13.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:13.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:13.372 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T14:59:13.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:59:13.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:13.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:59:14.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:15.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:15.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:15.017 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T14:59:15.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:59:15.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:15.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:59:15.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:16.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:16.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:16.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T14:59:16.458 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T14:59:16.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:16.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:59:16.774 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/173272,v1:127.0.0.1:6811/173272] [v2:127.0.0.1:6812/173272,v1:127.0.0.1:6813/173272] exists,up e3412a06-2285-4d0b-af4a-8086931e9f3d 2024-09-18T14:59:16.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:59:16.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:59:16.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:59:16.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-09-18T14:59:16.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:615: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:59:16.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T14:59:16.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:59:16.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T14:59:16.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:59:16.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-18T14:59:16.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T14:59:16.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:59:16.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:59:16.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:59:16.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T14:59:16.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T14:59:16.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:59:16.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:59:16.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T14:59:16.779 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:59:16.779 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:59:16.779 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:59:16.779 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:59:16.779 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:59:16.779 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T14:59:16.781 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:59:16.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T14:59:16.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:59:16.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:59:16.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:59:16.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T14:59:16.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T14:59:16.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:59:16.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:59:16.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:59:16.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:59:16.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:59:16.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T14:59:16.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-18T14:59:16.782 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:59:16.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=44e2959f-193d-45d2-b1d9-31c4d42ffcbe 2024-09-18T14:59:16.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 44e2959f-193d-45d2-b1d9-31c4d42ffcbe' 2024-09-18T14:59:16.783 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 44e2959f-193d-45d2-b1d9-31c4d42ffcbe 2024-09-18T14:59:16.783 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:59:16.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDE6upmWruLLxAAT8nkk/A3JNdN6lF8+ej0UA== 2024-09-18T14:59:16.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDE6upmWruLLxAAT8nkk/A3JNdN6lF8+ej0UA=="}' 2024-09-18T14:59:16.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 44e2959f-193d-45d2-b1d9-31c4d42ffcbe -i td/osd-scrub-repair/2/new.json 2024-09-18T14:59:17.292 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:59:17.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-18T14:59:17.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDE6upmWruLLxAAT8nkk/A3JNdN6lF8+ej0UA== --osd-uuid 44e2959f-193d-45d2-b1d9-31c4d42ffcbe 2024-09-18T14:59:17.348 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:59:17.343+0000 7f9d1f15a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:59:17.351 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:59:17.343+0000 7f9d1f15a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:59:17.353 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:59:17.347+0000 7f9d1f15a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:59:17.353 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:59:17.347+0000 7f9d1f15a540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T14:59:17.353 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:59:17.347+0000 7f9d1f15a540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T14:59:17.353 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:59:17.347+0000 7f9d1f15a540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T14:59:17.420 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:59:17.415+0000 7f9d1f15a540 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-18T14:59:24.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-18T14:59:24.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:59:24.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T14:59:24.111 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T14:59:24.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:59:24.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T14:59:24.493 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T14:59:24.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T14:59:24.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:59:24.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:59:24.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:59:24.525 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:59:24.519+0000 7fafbdf59540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:59:24.527 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:59:24.519+0000 7fafbdf59540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:59:24.528 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:59:24.523+0000 7fafbdf59540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:59:24.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T14:59:24.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:59:24.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:59:24.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:59:24.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:59:24.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:24.856 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T14:59:24.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:59:24.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:24.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:59:25.094 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:59:25.087+0000 7fafbdf59540 -1 Falling back to public interface 2024-09-18T14:59:25.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:26.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:26.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:26.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:59:26.210 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T14:59:26.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:26.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:59:26.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:27.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:27.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:27.518 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T14:59:27.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:59:27.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:27.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:59:27.630 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T14:59:27.623+0000 7fafbdf59540 -1 osd.2 0 log_to_monitors true 2024-09-18T14:59:27.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:28.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:28.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:28.852 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T14:59:28.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:59:28.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:28.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:59:29.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:30.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:30.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:30.168 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T14:59:30.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:59:30.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:30.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:59:30.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:31.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:31.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:31.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:59:31.574 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T14:59:31.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:31.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:59:31.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:32.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:32.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:32.891 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T14:59:32.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:59:32.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:32.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:59:33.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:34.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:34.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:34.211 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T14:59:34.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:59:34.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:34.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:59:34.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:35.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:35.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:35.611 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T14:59:35.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:59:35.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:35.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:59:35.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:36.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:36.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:36.919 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T14:59:36.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:59:36.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:36.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:59:37.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:38.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:38.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:38.272 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T14:59:38.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:59:38.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:38.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:59:38.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:39.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:39.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:39.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:59:39.737 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T14:59:39.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:39.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:59:40.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:41.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:41.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:41.051 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T14:59:41.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:59:41.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:41.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:59:41.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:42.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:42.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:42.484 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T14:59:42.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:59:42.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:42.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:59:43.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:44.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:44.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:44.004 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T14:59:44.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:59:44.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:44.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:59:44.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:45.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:45.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:45.448 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T14:59:45.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:59:45.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:45.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:59:46.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:47.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:47.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:47.112 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T14:59:47.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:59:47.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:47.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:59:47.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:48.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:48.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:48.718 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T14:59:48.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:59:48.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:48.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:59:49.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:50.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:50.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:50.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:59:50.248 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T14:59:50.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:50.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:59:50.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:51.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:51.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:51.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T14:59:51.569 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T14:59:51.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:51.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:59:52.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:59:53.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:59:53.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:59:53.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-18T14:59:53.012 INFO:tasks.workunit.client.0.smithi046.stdout:20 2024-09-18T14:59:53.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:59:53.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:59:53.329 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/175077,v1:127.0.0.1:6819/175077] [v2:127.0.0.1:6820/175077,v1:127.0.0.1:6821/175077] exists,up 44e2959f-193d-45d2-b1d9-31c4d42ffcbe 2024-09-18T14:59:53.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:59:53.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:59:53.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:59:53.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed_norecov: create_pool testpool 1 1 2024-09-18T14:59:53.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-09-18T14:59:53.840 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'testpool' created 2024-09-18T14:59:53.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:59:54.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:619: TEST_auto_repair_bluestore_failed_norecov: ceph osd pool set testpool size 2 2024-09-18T14:59:55.384 INFO:tasks.workunit.client.0.smithi046.stderr:set pool 1 size to 2 2024-09-18T14:59:55.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-09-18T14:59:55.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:59:55.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:59:55.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:59:55.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T14:59:55.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:59:55.399 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:59:55.399 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:59:55.399 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:59:55.400 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:59:55.400 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:59:55.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:59:55.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:59:55.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:59:55.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:59:55.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:59:55.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:59:55.990 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T14:59:55.990 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T14:59:55.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:59:55.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:59:55.990 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:59:56.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836507 2024-09-18T14:59:56.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836507 2024-09-18T14:59:56.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507' 2024-09-18T14:59:56.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:59:56.140 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:59:56.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672975 2024-09-18T14:59:56.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672975 2024-09-18T14:59:56.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507 1-42949672975' 2024-09-18T14:59:56.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:59:56.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:59:56.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509449 2024-09-18T14:59:56.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509449 2024-09-18T14:59:56.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507 1-42949672975 2-64424509449' 2024-09-18T14:59:56.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:59:56.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836507 2024-09-18T14:59:56.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:59:56.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:59:56.428 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836507 2024-09-18T14:59:56.429 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:59:56.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836507 2024-09-18T14:59:56.431 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836507 2024-09-18T14:59:56.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-18T14:59:56.432 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:59:56.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836506 -lt 21474836507 2024-09-18T14:59:56.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:59:57.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:59:57.747 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:59:58.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836508 -lt 21474836507 2024-09-18T14:59:58.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:59:58.068 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672975 2024-09-18T14:59:58.068 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:59:58.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:59:58.071 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672975 2024-09-18T14:59:58.071 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:59:58.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672975 2024-09-18T14:59:58.073 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672975 2024-09-18T14:59:58.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-18T14:59:58.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:59:58.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672976 -lt 42949672975 2024-09-18T14:59:58.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:59:58.423 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509449 2024-09-18T14:59:58.424 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:59:58.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:59:58.427 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509449 2024-09-18T14:59:58.427 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:59:58.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509449 2024-09-18T14:59:58.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-18T14:59:58.430 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509449 2024-09-18T14:59:58.430 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:59:58.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509449 -lt 64424509449 2024-09-18T14:59:58.780 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:59:58.781 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:59:58.781 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:59:59.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T14:59:59.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:59:59.186 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:59:59.186 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:59:59.186 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:59:59.186 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:59:59.186 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:59:59.187 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:59:59.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T14:59:59.507 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:59:59.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:59:59.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:59:59.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T14:59:59.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:59:59.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:59:59.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: local payload=ABCDEF 2024-09-18T14:59:59.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: echo ABCDEF 2024-09-18T14:59:59.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:625: TEST_auto_repair_bluestore_failed_norecov: seq 1 10 2024-09-18T14:59:59.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:625: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-18T14:59:59.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:627: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:00:00.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:625: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-18T15:00:00.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:627: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:00:00.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:625: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-18T15:00:00.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:627: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:00:00.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:625: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-18T15:00:00.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:627: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:00:00.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:625: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-18T15:00:00.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:627: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:00:00.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:625: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-18T15:00:00.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:627: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:00:00.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:625: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-18T15:00:00.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:627: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:00:00.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:625: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-18T15:00:00.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:627: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:00:00.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:625: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-18T15:00:00.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:627: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:00:01.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:625: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-18T15:00:01.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:627: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:00:01.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-09-18T15:00:01.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-09-18T15:00:01.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-09-18T15:00:01.308 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-09-18T15:00:01.308 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-09-18T15:00:01.308 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T15:00:01.309 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T15:00:01.309 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:00:01.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T15:00:01.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T15:00:01.629 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T15:00:02.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-09-18T15:00:02.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:00:02.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:00:02.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:00:02.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:00:02.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-09-18T15:00:02.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:00:02.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:00:02.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:00:02.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:00:02.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:00:02.013 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:00:02.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:00:02.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:00:02.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:00:02.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:00:02.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:00:02.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-09-18T15:00:02.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:00:02.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:00:02.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:00:02.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:00:02.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:00:02.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-09-18T15:00:03.970 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:ff7b1f36:::obj1:head# 2024-09-18T15:00:04.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:00:04.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:00:04.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:00:04.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:00:04.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:00:04.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:00:04.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:00:04.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:00:04.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:00:04.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:00:04.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:00:04.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:00:04.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:00:04.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:00:04.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:00:04.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:00:04.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:00:04.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:00:04.510 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:00:04.510 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:00:04.510 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:00:04.510 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:00:04.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:00:04.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:00:04.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:00:04.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:00:04.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:00:04.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:00:04.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:00:04.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:00:04.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:00:04.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:00:04.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:00:04.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:00:04.511 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:00:04.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:00:04.513 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:00:04.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:00:04.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:00:04.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:00:04.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:00:04.549 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:00:04.542+0000 7fa7e3688540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:00:04.551 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:00:04.546+0000 7fa7e3688540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:00:04.554 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:00:04.546+0000 7fa7e3688540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:00:04.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:00:04.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:00:04.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:00:04.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:00:04.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:00:04.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:00:04.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:00:04.833 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:00:04.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:00:04.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:00:05.086 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:00:05.078+0000 7fa7e3688540 -1 Falling back to public interface 2024-09-18T15:00:05.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:00:06.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:00:06.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:00:06.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:00:06.177 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:00:06.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:00:06.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:00:06.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:00:07.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:00:07.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:00:07.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:00:07.487 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:00:07.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:00:07.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:00:07.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:00:08.023 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:00:08.014+0000 7fa7e3688540 -1 osd.0 21 log_to_monitors true 2024-09-18T15:00:08.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:00:08.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:00:08.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:00:08.807 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:00:08.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:00:08.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:00:09.143 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/177156,v1:127.0.0.1:6803/177156] [v2:127.0.0.1:6804/177156,v1:127.0.0.1:6805/177156] exists,up 33c23184-ae55-4b17-9812-4bddeeecfc1d 2024-09-18T15:00:09.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:00:09.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:00:09.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:00:09.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:00:09.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:00:09.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:00:09.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:00:09.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:00:09.144 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:00:09.145 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:00:09.145 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:00:09.145 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:00:09.145 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:00:09.145 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:00:09.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:00:09.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:00:09.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:00:09.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:00:09.366 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:00:09.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:00:09.680 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:00:09.680 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:00:09.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:00:09.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:00:09.680 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:00:09.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182402 2024-09-18T15:00:09.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182402 2024-09-18T15:00:09.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182402' 2024-09-18T15:00:09.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:00:09.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:00:09.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672979 2024-09-18T15:00:09.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672979 2024-09-18T15:00:09.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182402 1-42949672979' 2024-09-18T15:00:09.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:00:09.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:00:10.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509453 2024-09-18T15:00:10.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509453 2024-09-18T15:00:10.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182402 1-42949672979 2-64424509453' 2024-09-18T15:00:10.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:00:10.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-107374182402 2024-09-18T15:00:10.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:00:10.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:00:10.124 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-107374182402 2024-09-18T15:00:10.124 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:00:10.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182402 2024-09-18T15:00:10.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-09-18T15:00:10.125 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 107374182402 2024-09-18T15:00:10.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:00:10.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 107374182402 2024-09-18T15:00:10.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:00:11.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:00:11.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:00:11.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182402 -lt 107374182402 2024-09-18T15:00:11.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:00:11.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672979 2024-09-18T15:00:11.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:00:11.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:00:11.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672979 2024-09-18T15:00:11.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:00:11.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672979 2024-09-18T15:00:11.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-18T15:00:11.763 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 42949672979 2024-09-18T15:00:11.763 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:00:12.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672979 -lt 42949672979 2024-09-18T15:00:12.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:00:12.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509453 2024-09-18T15:00:12.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:00:12.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:00:12.125 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509453 2024-09-18T15:00:12.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:00:12.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509453 2024-09-18T15:00:12.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-18T15:00:12.127 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509453 2024-09-18T15:00:12.127 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:00:12.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509453 -lt 64424509453 2024-09-18T15:00:12.464 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:00:12.464 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:00:12.464 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:00:12.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:00:12.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:00:12.824 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:00:12.825 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:00:12.825 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:00:12.825 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:00:12.825 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:00:12.825 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:00:13.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:00:13.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:00:13.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:00:13.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:00:13.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:00:13.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:00:13.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:00:13.543 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:634: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-09-18T15:00:13.543 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-09-18T15:00:13.543 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T15:00:13.544 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T15:00:13.544 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:00:13.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:634: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj1 rm-attr _ 2024-09-18T15:00:13.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:00:13.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:00:13.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:00:13.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:00:13.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj1 rm-attr _ 2024-09-18T15:00:13.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:00:13.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:00:13.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:00:13.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:00:13.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:00:13.890 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:00:13.890 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:00:13.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:00:13.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:00:13.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:00:14.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:00:14.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 rm-attr _ 2024-09-18T15:00:14.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:00:14.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:00:14.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:00:14.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:00:14.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:00:14.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 rm-attr _ 2024-09-18T15:00:16.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:00:16.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:00:16.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:00:16.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:00:16.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:00:16.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:00:16.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:00:16.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:00:16.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:00:16.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:00:16.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:00:16.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:00:16.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:00:16.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:00:16.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:00:16.301 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:00:16.301 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:00:16.301 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:00:16.301 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:00:16.301 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:00:16.301 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:00:16.301 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:00:16.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:00:16.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:00:16.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:00:16.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:00:16.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:00:16.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:00:16.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:00:16.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:00:16.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:00:16.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:00:16.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:00:16.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:00:16.304 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:00:16.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:00:16.305 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:00:16.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:00:16.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:00:16.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:00:16.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:00:16.338 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:00:16.330+0000 7f5f90e8e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:00:16.342 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:00:16.334+0000 7f5f90e8e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:00:16.344 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:00:16.338+0000 7f5f90e8e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:00:16.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:00:16.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:00:16.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:00:16.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:00:16.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:00:16.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:00:16.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:00:16.626 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:00:16.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:00:16.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:00:16.897 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:00:16.890+0000 7f5f90e8e540 -1 Falling back to public interface 2024-09-18T15:00:16.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:00:17.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:00:17.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:00:17.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:00:17.980 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:00:17.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:00:17.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:00:18.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:00:19.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:00:19.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:00:19.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:00:19.291 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:00:19.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:00:19.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:00:19.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:00:19.836 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:00:19.830+0000 7f5f90e8e540 -1 osd.1 26 log_to_monitors true 2024-09-18T15:00:20.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:00:20.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:00:20.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:00:20.614 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:00:20.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:00:20.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:00:20.966 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 30 up_thru 30 down_at 27 last_clean_interval [10,26) [v2:127.0.0.1:6810/178535,v1:127.0.0.1:6811/178535] [v2:127.0.0.1:6812/178535,v1:127.0.0.1:6813/178535] exists,up e3412a06-2285-4d0b-af4a-8086931e9f3d 2024-09-18T15:00:20.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:00:20.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:00:20.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:00:20.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:00:20.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:00:20.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:00:20.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:00:20.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:00:20.967 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:00:20.968 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:00:20.968 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:00:20.968 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:00:20.969 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:00:20.969 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:00:21.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:00:21.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:00:21.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:00:21.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:00:21.185 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:00:21.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:00:21.512 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:00:21.512 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:00:21.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:00:21.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:00:21.513 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:00:21.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182405 2024-09-18T15:00:21.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182405 2024-09-18T15:00:21.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182405' 2024-09-18T15:00:21.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:00:21.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:00:21.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018882 2024-09-18T15:00:21.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018882 2024-09-18T15:00:21.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182405 1-128849018882' 2024-09-18T15:00:21.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:00:21.919 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:00:22.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509456 2024-09-18T15:00:22.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509456 2024-09-18T15:00:22.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182405 1-128849018882 2-64424509456' 2024-09-18T15:00:22.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:00:22.107 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-107374182405 2024-09-18T15:00:22.107 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:00:22.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:00:22.111 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-107374182405 2024-09-18T15:00:22.111 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:00:22.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182405 2024-09-18T15:00:22.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 107374182405' 2024-09-18T15:00:22.112 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 107374182405 2024-09-18T15:00:22.113 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:00:22.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182404 -lt 107374182405 2024-09-18T15:00:22.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:00:23.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:00:23.430 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:00:23.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182405 -lt 107374182405 2024-09-18T15:00:23.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:00:23.746 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-128849018882 2024-09-18T15:00:23.747 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:00:23.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:00:23.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-128849018882 2024-09-18T15:00:23.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:00:23.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018882 2024-09-18T15:00:23.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 128849018882' 2024-09-18T15:00:23.752 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 128849018882 2024-09-18T15:00:23.753 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:00:24.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018882 -lt 128849018882 2024-09-18T15:00:24.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:00:24.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509456 2024-09-18T15:00:24.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:00:24.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:00:24.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509456 2024-09-18T15:00:24.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:00:24.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509456 2024-09-18T15:00:24.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-09-18T15:00:24.099 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509456 2024-09-18T15:00:24.100 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:00:24.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509456 -lt 64424509456 2024-09-18T15:00:24.400 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:00:24.400 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:00:24.400 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:00:24.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:00:24.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:00:24.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:00:24.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:00:24.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:00:24.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:00:24.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:00:24.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:00:25.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:00:25.167 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:00:25.168 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:00:25.168 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:00:25.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:00:25.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:00:25.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:00:25.537 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-09-18T15:00:25.537 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-09-18T15:00:25.538 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-09-18T15:00:25.538 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-09-18T15:00:25.538 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-09-18T15:00:25.538 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T15:00:25.538 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T15:00:25.538 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:00:25.883 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T15:00:25.883 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T15:00:25.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T15:00:26.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-09-18T15:00:26.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:00:26.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:00:26.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:00:26.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:00:26.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-09-18T15:00:26.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:00:26.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:00:26.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:00:26.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:00:26.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:00:26.187 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:00:26.187 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:00:26.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:00:26.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:00:26.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:00:26.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:00:26.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-09-18T15:00:26.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:00:26.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:00:26.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:00:26.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:00:26.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:00:26.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-09-18T15:00:28.084 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:104778fc:::obj2:head# 2024-09-18T15:00:28.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:00:28.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:00:28.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:00:28.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:00:28.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:00:28.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:00:28.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:00:28.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:00:28.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:00:28.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:00:28.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:00:28.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:00:28.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:00:28.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:00:28.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:00:28.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:00:28.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:00:28.625 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:00:28.625 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:00:28.625 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:00:28.625 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:00:28.625 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:00:28.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:00:28.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:00:28.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:00:28.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:00:28.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:00:28.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:00:28.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:00:28.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:00:28.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:00:28.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:00:28.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:00:28.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:00:28.627 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:00:28.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:00:28.629 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:00:28.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:00:28.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:00:28.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:00:28.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:00:28.662 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:00:28.654+0000 7f8ef822d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:00:28.666 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:00:28.658+0000 7f8ef822d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:00:28.671 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:00:28.662+0000 7f8ef822d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:00:28.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:00:28.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:00:28.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:00:28.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:00:28.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:00:28.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:00:28.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:00:28.933 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:00:28.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:00:28.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:00:29.205 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:00:29.198+0000 7f8ef822d540 -1 Falling back to public interface 2024-09-18T15:00:29.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:00:30.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:00:30.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:00:30.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:00:30.283 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:00:30.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:00:30.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:00:30.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:00:31.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:00:31.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:00:31.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:00:31.602 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:00:31.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:00:31.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:00:31.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:00:31.972 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:00:31.966+0000 7f8ef822d540 -1 osd.0 31 log_to_monitors true 2024-09-18T15:00:32.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:00:32.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:00:32.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:00:32.921 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:00:32.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:00:32.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:00:33.381 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 35 up_thru 27 down_at 32 last_clean_interval [25,31) [v2:127.0.0.1:6802/179939,v1:127.0.0.1:6803/179939] [v2:127.0.0.1:6804/179939,v1:127.0.0.1:6805/179939] exists,up 33c23184-ae55-4b17-9812-4bddeeecfc1d 2024-09-18T15:00:33.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:00:33.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:00:33.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:00:33.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:00:33.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:00:33.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:00:33.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:00:33.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:00:33.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:00:33.382 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:00:33.382 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:00:33.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:00:33.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:00:33.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:00:33.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:00:33.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:00:33.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:00:33.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:00:33.604 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:00:33.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:00:33.917 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:00:33.917 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:00:33.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:00:33.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:00:33.917 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:00:34.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855362 2024-09-18T15:00:34.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855362 2024-09-18T15:00:34.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855362' 2024-09-18T15:00:34.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:00:34.068 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:00:34.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018885 2024-09-18T15:00:34.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018885 2024-09-18T15:00:34.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855362 1-128849018885' 2024-09-18T15:00:34.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:00:34.204 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:00:34.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509460 2024-09-18T15:00:34.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509460 2024-09-18T15:00:34.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855362 1-128849018885 2-64424509460' 2024-09-18T15:00:34.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:00:34.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-150323855362 2024-09-18T15:00:34.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:00:34.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:00:34.348 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-150323855362 2024-09-18T15:00:34.348 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:00:34.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855362 2024-09-18T15:00:34.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-09-18T15:00:34.351 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 150323855362 2024-09-18T15:00:34.352 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:00:34.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 150323855362 2024-09-18T15:00:34.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:00:35.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:00:35.664 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:00:35.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855362 -lt 150323855362 2024-09-18T15:00:35.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:00:35.978 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-128849018885 2024-09-18T15:00:35.979 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:00:35.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:00:35.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-128849018885 2024-09-18T15:00:35.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:00:35.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018885 2024-09-18T15:00:35.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 128849018885' 2024-09-18T15:00:35.984 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 128849018885 2024-09-18T15:00:35.984 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:00:36.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018885 -lt 128849018885 2024-09-18T15:00:36.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:00:36.294 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509460 2024-09-18T15:00:36.294 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:00:36.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:00:36.297 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509460 2024-09-18T15:00:36.298 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:00:36.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509460 2024-09-18T15:00:36.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-09-18T15:00:36.300 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509460 2024-09-18T15:00:36.300 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:00:36.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509460 -lt 64424509460 2024-09-18T15:00:36.645 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:00:36.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:00:36.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:00:37.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:00:37.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:00:37.010 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:00:37.010 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:00:37.010 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:00:37.010 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:00:37.011 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:00:37.011 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:00:37.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:00:37.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:00:37.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:00:37.380 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:00:37.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:00:37.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:00:37.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:00:37.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:637: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-09-18T15:00:37.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-09-18T15:00:37.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T15:00:37.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T15:00:37.744 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:00:38.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:637: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-18T15:00:38.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:00:38.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:00:38.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:00:38.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:00:38.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-18T15:00:38.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:00:38.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:00:38.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:00:38.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:00:38.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:00:38.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:00:38.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:00:38.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:00:38.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:00:38.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:00:38.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:00:38.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-18T15:00:38.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:00:38.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:00:38.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:00:38.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:00:38.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:00:38.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-09-18T15:00:39.886 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:ff7b1f36:::obj1:head#, (61) No data available 2024-09-18T15:00:40.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:00:40.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:00:40.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:00:40.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:00:40.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:00:40.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:00:40.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:00:40.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:00:40.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:00:40.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:00:40.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:00:40.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:00:40.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:00:40.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:00:40.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:00:40.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:00:40.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:00:40.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:00:40.427 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:00:40.427 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:00:40.427 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:00:40.427 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:00:40.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:00:40.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:00:40.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:00:40.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:00:40.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:00:40.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:00:40.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:00:40.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:00:40.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:00:40.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:00:40.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:00:40.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:00:40.428 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:00:40.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:00:40.430 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:00:40.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:00:40.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:00:40.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:00:40.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:00:40.462 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:00:40.454+0000 7f09a531e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:00:40.468 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:00:40.458+0000 7f09a531e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:00:40.475 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:00:40.458+0000 7f09a531e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:00:40.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:00:40.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:00:40.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:00:40.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:00:40.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:00:40.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:00:40.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:00:40.744 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:00:40.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:00:40.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:00:41.001 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:00:40.994+0000 7f09a531e540 -1 Falling back to public interface 2024-09-18T15:00:41.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:00:42.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:00:42.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:00:42.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:00:42.055 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:00:42.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:00:42.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:00:42.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:00:43.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:00:43.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:00:43.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:00:43.369 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:00:43.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:00:43.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:00:43.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:00:43.883 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:00:43.874+0000 7f09a531e540 -1 osd.1 36 log_to_monitors true 2024-09-18T15:00:44.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:00:44.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:00:44.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:00:44.729 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:00:44.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:00:44.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:00:45.177 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6810/181318,v1:127.0.0.1:6811/181318] [v2:127.0.0.1:6812/181318,v1:127.0.0.1:6813/181318] exists,up e3412a06-2285-4d0b-af4a-8086931e9f3d 2024-09-18T15:00:45.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:00:45.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:00:45.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:00:45.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:00:45.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:00:45.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:00:45.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:00:45.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:00:45.179 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:00:45.180 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:00:45.180 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:00:45.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:00:45.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:00:45.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:00:45.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:00:45.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:00:45.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:00:45.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:00:45.403 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:00:45.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:00:45.715 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:00:45.715 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:00:45.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:00:45.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:00:45.715 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:00:45.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855365 2024-09-18T15:00:45.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855365 2024-09-18T15:00:45.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855365' 2024-09-18T15:00:45.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:00:45.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:00:46.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691842 2024-09-18T15:00:46.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691842 2024-09-18T15:00:46.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855365 1-171798691842' 2024-09-18T15:00:46.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:00:46.084 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:00:46.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509463 2024-09-18T15:00:46.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509463 2024-09-18T15:00:46.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855365 1-171798691842 2-64424509463' 2024-09-18T15:00:46.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:00:46.229 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-150323855365 2024-09-18T15:00:46.230 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:00:46.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:00:46.233 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-150323855365 2024-09-18T15:00:46.233 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:00:46.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855365 2024-09-18T15:00:46.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 150323855365' 2024-09-18T15:00:46.234 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 150323855365 2024-09-18T15:00:46.235 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:00:46.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855364 -lt 150323855365 2024-09-18T15:00:46.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:00:47.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:00:47.546 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:00:47.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855365 -lt 150323855365 2024-09-18T15:00:47.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:00:47.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-171798691842 2024-09-18T15:00:47.864 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:00:47.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:00:47.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-171798691842 2024-09-18T15:00:47.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:00:47.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691842 2024-09-18T15:00:47.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2024-09-18T15:00:47.869 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 171798691842 2024-09-18T15:00:47.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:00:48.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691842 -lt 171798691842 2024-09-18T15:00:48.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:00:48.227 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509463 2024-09-18T15:00:48.227 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:00:48.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:00:48.231 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509463 2024-09-18T15:00:48.231 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:00:48.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509463 2024-09-18T15:00:48.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-09-18T15:00:48.233 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509463 2024-09-18T15:00:48.233 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:00:48.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509463 -lt 64424509463 2024-09-18T15:00:48.574 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:00:48.574 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:00:48.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:00:48.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:00:48.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:00:48.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:00:48.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:00:48.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:00:48.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:00:48.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:00:48.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:00:49.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:00:49.352 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:00:49.353 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:00:49.353 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:00:49.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:00:49.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:00:49.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:00:49.719 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: get_pg testpool obj1 2024-09-18T15:00:49.719 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-09-18T15:00:49.719 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-09-18T15:00:49.719 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-09-18T15:00:49.720 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:00:50.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: local pgid=1.0 2024-09-18T15:00:50.080 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool obj1 2024-09-18T15:00:50.080 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-09-18T15:00:50.080 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:00:50.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-09-18T15:00:50.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:00:50.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: local primary=1 2024-09-18T15:00:50.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: get_last_scrub_stamp 1.0 2024-09-18T15:00:50.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:00:50.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:00:50.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:00:50.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:00:50.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: local last_scrub_stamp=2024-09-18T14:59:53.725932+0000 2024-09-18T15:00:50.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:642: TEST_auto_repair_bluestore_failed_norecov: ceph tell 1.0 deep_scrub 2024-09-18T15:00:50.867 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T15:00:50.867 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": true, 2024-09-18T15:00:50.867 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:59:10.863150+0000" 2024-09-18T15:00:50.867 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T15:00:50.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:643: TEST_auto_repair_bluestore_failed_norecov: ceph tell 1.0 scrub 2024-09-18T15:00:51.056 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T15:00:51.057 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T15:00:51.057 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T14:59:11.052489+0000" 2024-09-18T15:00:51.057 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T15:00:51.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:646: TEST_auto_repair_bluestore_failed_norecov: wait_for_scrub 1.0 2024-09-18T14:59:53.725932+0000 2024-09-18T15:00:51.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-18T15:00:51.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T14:59:53.725932+0000 2024-09-18T15:00:51.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:00:51.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:00:51.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:00:51.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T15:00:51.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:00:51.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:00:51.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:00:51.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:00:51.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:59:53.725932+0000 '>' 2024-09-18T14:59:53.725932+0000 2024-09-18T15:00:51.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:00:52.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:00:52.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:00:52.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T15:00:52.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:00:52.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:00:52.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:00:52.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:00:52.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:59:53.725932+0000 '>' 2024-09-18T14:59:53.725932+0000 2024-09-18T15:00:52.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:00:53.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:00:53.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:00:53.699 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T15:00:53.699 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:00:53.699 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:00:53.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:00:53.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:00:54.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T14:59:53.725932+0000 '>' 2024-09-18T14:59:53.725932+0000 2024-09-18T15:00:54.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:00:55.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:00:55.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:00:55.015 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T15:00:55.015 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:00:55.015 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:00:55.016 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:00:55.016 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:00:55.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:00:52.936551+0000 '>' 2024-09-18T14:59:53.725932+0000 2024-09-18T15:00:55.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:00:55.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:647: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-09-18T15:00:55.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:00:55.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:00:55.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:00:55.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:00:55.328 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:00:55.328 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:00:55.329 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:00:55.329 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:00:55.329 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:00:55.329 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:00:55.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:00:55.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:00:55.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:00:55.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:00:55.553 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:00:55.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:00:55.862 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:00:55.862 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:00:55.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:00:55.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:00:55.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:00:56.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855368 2024-09-18T15:00:56.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855368 2024-09-18T15:00:56.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855368' 2024-09-18T15:00:56.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:00:56.004 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:00:56.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691845 2024-09-18T15:00:56.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691845 2024-09-18T15:00:56.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855368 1-171798691845' 2024-09-18T15:00:56.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:00:56.143 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:00:56.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509466 2024-09-18T15:00:56.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509466 2024-09-18T15:00:56.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855368 1-171798691845 2-64424509466' 2024-09-18T15:00:56.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:00:56.287 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-150323855368 2024-09-18T15:00:56.287 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:00:56.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:00:56.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-150323855368 2024-09-18T15:00:56.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:00:56.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855368 2024-09-18T15:00:56.293 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 150323855368 2024-09-18T15:00:56.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 150323855368' 2024-09-18T15:00:56.294 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:00:56.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855367 -lt 150323855368 2024-09-18T15:00:56.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:00:57.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:00:57.610 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:00:57.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855368 -lt 150323855368 2024-09-18T15:00:57.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:00:57.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-171798691845 2024-09-18T15:00:57.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:00:57.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:00:57.934 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-171798691845 2024-09-18T15:00:57.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:00:57.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691845 2024-09-18T15:00:57.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 171798691845' 2024-09-18T15:00:57.937 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 171798691845 2024-09-18T15:00:57.937 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:00:58.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691845 -lt 171798691845 2024-09-18T15:00:58.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:00:58.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509466 2024-09-18T15:00:58.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:00:58.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:00:58.285 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509466 2024-09-18T15:00:58.285 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:00:58.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509466 2024-09-18T15:00:58.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2024-09-18T15:00:58.287 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509466 2024-09-18T15:00:58.288 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:00:58.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509466 -lt 64424509466 2024-09-18T15:00:58.626 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:00:58.626 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:00:58.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:00:58.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:00:58.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:00:58.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:00:58.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:00:58.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:00:58.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:00:58.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:00:58.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:00:59.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:00:59.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:00:59.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:00:59.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:00:59.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:00:59.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:00:59.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:00:59.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:648: TEST_auto_repair_bluestore_failed_norecov: flush_pg_stats 2024-09-18T15:00:59.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:00:59.685 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:01:00.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:01:00.095 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:01:00.095 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:01:00.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:01:00.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:01:00.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:01:00.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855370 2024-09-18T15:01:00.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855370 2024-09-18T15:01:00.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855370' 2024-09-18T15:01:00.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:01:00.231 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:01:00.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691847 2024-09-18T15:01:00.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691847 2024-09-18T15:01:00.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855370 1-171798691847' 2024-09-18T15:01:00.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:01:00.367 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:01:00.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509468 2024-09-18T15:01:00.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509468 2024-09-18T15:01:00.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855370 1-171798691847 2-64424509468' 2024-09-18T15:01:00.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:01:00.511 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-150323855370 2024-09-18T15:01:00.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:01:00.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:01:00.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-150323855370 2024-09-18T15:01:00.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:01:00.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855370 2024-09-18T15:01:00.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 150323855370' 2024-09-18T15:01:00.516 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 150323855370 2024-09-18T15:01:00.517 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:01:00.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855369 -lt 150323855370 2024-09-18T15:01:00.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:01:01.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:01:01.825 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:01:02.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855370 -lt 150323855370 2024-09-18T15:01:02.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:01:02.140 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-171798691847 2024-09-18T15:01:02.141 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:01:02.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:01:02.144 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-171798691847 2024-09-18T15:01:02.144 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:01:02.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691847 2024-09-18T15:01:02.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 171798691847' 2024-09-18T15:01:02.146 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 171798691847 2024-09-18T15:01:02.146 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:01:02.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691847 -lt 171798691847 2024-09-18T15:01:02.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:01:02.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509468 2024-09-18T15:01:02.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:01:02.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:01:02.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:01:02.513 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509468 2024-09-18T15:01:02.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509468 2024-09-18T15:01:02.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-09-18T15:01:02.514 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509468 2024-09-18T15:01:02.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:01:02.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509468 -lt 64424509468 2024-09-18T15:01:02.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:649: TEST_auto_repair_bluestore_failed_norecov: grep -q 'scrub_finish.*present with no repair possible' td/osd-scrub-repair/osd.1.log 2024-09-18T15:01:02.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-09-18T15:01:03.195 INFO:tasks.workunit.client.0.smithi046.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T15:01:03.195 INFO:tasks.workunit.client.0.smithi046.stdout:1.0 10 0 0 0 0 56 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-09-18T15:00:52.936748+0000 21'10 41:109 [1,0] 1 [1,0] 1 21'10 2024-09-18T15:00:52.936551+0000 21'10 2024-09-18T15:00:52.936551+0000 0 1 periodic scrub scheduled @ 2024-09-19T15:00:52.936551+0000 10 0 2024-09-18T15:01:03.195 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-09-18T15:01:03.195 INFO:tasks.workunit.client.0.smithi046.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T15:01:03.199 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T15:01:03.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:651: TEST_auto_repair_bluestore_failed_norecov: grep -q '^1.0.*+failed_repair' 2024-09-18T15:01:03.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:651: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-09-18T15:01:03.499 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T15:01:03.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T15:01:03.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:01:03.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:01:03.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:01:03.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:01:03.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:01:03.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:01:03.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:01:03.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:01:03.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:01:03.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:01:03.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:01:03.629 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:01:03.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:01:03.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:01:03.632 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:01:03.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:01:03.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:01:03.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:01:03.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:01:03.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:01:03.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:01:03.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:01:03.661 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:01:03.661 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:01:03.662 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:01:03.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:01:03.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:01:03.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:01:03.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T15:01:03.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T15:01:03.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T15:01:03.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T15:01:03.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:01:03.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:01:03.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:01:03.665 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:01:03.665 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:01:03.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:01:03.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:01:03.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:01:03.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:01:03.670 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:01:03.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:01:03.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:01:03.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:01:03.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:01:03.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:01:03.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:01:03.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:01:03.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:01:03.677 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:01:03.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:01:03.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:01:03.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:01:03.684 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:01:03.684 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:01:03.684 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:01:03.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:01:03.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:01:03.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:01:03.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T15:01:03.688 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:01:03.688 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:01:03.689 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:01:03.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T15:01:03.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:01:03.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:01:03.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:01:03.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T15:01:03.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_auto_repair_bluestore_scrub td/osd-scrub-repair 2024-09-18T15:01:03.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:487: TEST_auto_repair_bluestore_scrub: local dir=td/osd-scrub-repair 2024-09-18T15:01:03.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:488: TEST_auto_repair_bluestore_scrub: local poolname=testpool 2024-09-18T15:01:03.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:491: TEST_auto_repair_bluestore_scrub: run_mon td/osd-scrub-repair a 2024-09-18T15:01:03.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T15:01:03.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:01:03.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:01:03.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:01:03.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T15:01:03.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-18T15:01:04.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:01:04.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:01:04.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:01:04.074 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:01:04.074 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:01:04.074 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:01:04.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:01:04.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T15:01:04.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:01:04.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:01:04.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:01:04.356 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:01:04.356 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:01:04.356 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:01:04.357 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:01:04.357 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:01:04.357 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:01:04.360 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:01:04.360 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:01:04.361 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:01:04.361 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:01:04.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:01:04.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T15:01:04.467 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:01:04.467 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:01:04.467 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:01:04.467 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:01:04.468 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:01:04.468 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:01:04.468 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:01:04.468 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:01:04.469 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:01:04.469 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:01:04.469 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:01:04.469 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:01:04.469 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:01:04.469 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T15:01:04.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:492: TEST_auto_repair_bluestore_scrub: run_mgr td/osd-scrub-repair x 2024-09-18T15:01:04.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T15:01:04.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:01:04.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:01:04.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:01:04.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T15:01:04.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:01:05.422 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:01:05.422 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:01:05.422 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:01:05.422 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:01:05.423 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:01:05.423 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:01:05.423 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:01:05.424 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:01:05.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:01:05.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:496: TEST_auto_repair_bluestore_scrub: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-18T15:01:05.462 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:497: TEST_auto_repair_bluestore_scrub: seq 0 2 2024-09-18T15:01:05.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:497: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-09-18T15:01:05.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-18T15:01:05.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:01:05.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:01:05.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:01:05.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:01:05.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:01:05.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:01:05.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:01:05.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:01:05.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:01:05.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:01:05.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:01:05.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:01:05.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:01:05.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:01:05.467 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:01:05.467 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:01:05.467 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:01:05.467 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:01:05.467 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:01:05.467 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:01:05.467 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:01:05.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:01:05.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:01:05.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:01:05.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:01:05.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:01:05.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:01:05.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:01:05.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:01:05.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:01:05.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:01:05.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:01:05.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-18T15:01:05.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:01:05.470 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:01:05.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4ff80dfa-aa33-4bc0-8200-c0feb377d9ec 2024-09-18T15:01:05.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4ff80dfa-aa33-4bc0-8200-c0feb377d9ec' 2024-09-18T15:01:05.472 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 4ff80dfa-aa33-4bc0-8200-c0feb377d9ec 2024-09-18T15:01:05.473 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:01:05.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAx6+pm7ucNHRAAZ6VM2DufbqEkvp29j24QgA== 2024-09-18T15:01:05.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAx6+pm7ucNHRAAZ6VM2DufbqEkvp29j24QgA=="}' 2024-09-18T15:01:05.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4ff80dfa-aa33-4bc0-8200-c0feb377d9ec -i td/osd-scrub-repair/0/new.json 2024-09-18T15:01:05.780 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:01:05.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-18T15:01:05.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQAx6+pm7ucNHRAAZ6VM2DufbqEkvp29j24QgA== --osd-uuid 4ff80dfa-aa33-4bc0-8200-c0feb377d9ec 2024-09-18T15:01:05.823 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:01:05.813+0000 7fdcc697d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:01:05.826 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:01:05.817+0000 7fdcc697d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:01:05.828 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:01:05.821+0000 7fdcc697d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:01:05.830 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:01:05.821+0000 7fdcc697d540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:01:05.830 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:01:05.821+0000 7fdcc697d540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:01:05.830 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:01:05.821+0000 7fdcc697d540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:01:05.913 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:01:05.905+0000 7fdcc697d540 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-18T15:01:12.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T15:01:12.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:01:12.055 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T15:01:12.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:01:12.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:01:12.439 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T15:01:12.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:01:12.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-18T15:01:12.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:01:12.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:01:12.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:01:12.471 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:01:12.461+0000 7f2d3e4c5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:01:12.474 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:01:12.465+0000 7f2d3e4c5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:01:12.475 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:01:12.465+0000 7f2d3e4c5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:01:12.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:01:12.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:01:12.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:01:12.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:01:12.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:01:12.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:12.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:01:12.743 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:01:12.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:12.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:01:12.758 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:01:12.749+0000 7f2d3e4c5540 -1 Falling back to public interface 2024-09-18T15:01:13.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:14.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:14.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:14.085 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:01:14.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:01:14.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:14.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:01:14.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:15.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:15.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:15.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:01:15.407 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:01:15.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:15.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:01:15.475 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:01:15.465+0000 7f2d3e4c5540 -1 osd.0 0 log_to_monitors true 2024-09-18T15:01:15.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:16.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:16.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:16.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:01:16.780 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:01:16.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:16.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:01:17.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:18.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:18.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:18.091 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:01:18.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:01:18.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:18.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:01:18.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:19.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:19.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:19.424 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:01:19.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:01:19.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:19.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:01:19.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:20.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:20.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:20.732 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:01:20.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:01:20.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:20.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:01:21.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:22.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:22.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:22.180 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:01:22.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:01:22.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:22.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:01:22.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:23.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:23.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:23.495 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:01:23.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:01:23.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:23.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:01:23.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:24.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:24.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:24.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:01:24.809 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:01:24.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:24.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:01:25.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:26.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:26.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:26.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:01:26.120 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:01:26.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:26.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:01:26.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:27.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:27.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:27.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:01:27.444 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:01:27.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:27.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:01:27.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:28.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:28.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:28.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:01:28.760 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:01:28.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:28.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:01:29.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:30.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:30.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:30.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:01:30.303 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:01:30.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:30.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:01:30.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:31.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:31.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:31.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:01:31.792 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:01:31.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:31.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:01:32.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:33.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:33.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:33.568 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:01:33.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:01:33.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:33.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:01:33.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:34.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:34.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:34.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:01:34.998 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:01:34.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:34.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:01:35.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:36.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:36.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:36.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:01:36.628 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:01:36.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:36.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:01:37.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:38.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:38.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:38.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:01:38.464 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:01:38.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:38.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:01:38.824 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/184266,v1:127.0.0.1:6803/184266] [v2:127.0.0.1:6804/184266,v1:127.0.0.1:6805/184266] exists,up 4ff80dfa-aa33-4bc0-8200-c0feb377d9ec 2024-09-18T15:01:38.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:01:38.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:01:38.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:01:38.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:497: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-09-18T15:01:38.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-18T15:01:38.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:01:38.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:01:38.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:01:38.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:01:38.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:01:38.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:01:38.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:01:38.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:01:38.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:01:38.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:01:38.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:01:38.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:01:38.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:01:38.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:01:38.828 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:01:38.829 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:01:38.829 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:01:38.830 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:01:38.831 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:01:38.831 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:01:38.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:01:38.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:01:38.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:01:38.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:01:38.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:01:38.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:01:38.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:01:38.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:01:38.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:01:38.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:01:38.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:01:38.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:01:38.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-18T15:01:38.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:01:38.832 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:01:38.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fe030fbb-d4c2-4e9c-b172-7695dc6417ed 2024-09-18T15:01:38.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 fe030fbb-d4c2-4e9c-b172-7695dc6417ed' 2024-09-18T15:01:38.834 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 fe030fbb-d4c2-4e9c-b172-7695dc6417ed 2024-09-18T15:01:38.835 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:01:38.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBS6+pm7lyrMhAAwbCbCyZxVrUap0rEIHLTYg== 2024-09-18T15:01:38.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBS6+pm7lyrMhAAwbCbCyZxVrUap0rEIHLTYg=="}' 2024-09-18T15:01:38.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fe030fbb-d4c2-4e9c-b172-7695dc6417ed -i td/osd-scrub-repair/1/new.json 2024-09-18T15:01:39.236 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:01:39.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-18T15:01:39.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQBS6+pm7lyrMhAAwbCbCyZxVrUap0rEIHLTYg== --osd-uuid fe030fbb-d4c2-4e9c-b172-7695dc6417ed 2024-09-18T15:01:39.291 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:01:39.281+0000 7f2ec3e7d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:01:39.296 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:01:39.285+0000 7f2ec3e7d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:01:39.296 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:01:39.289+0000 7f2ec3e7d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:01:39.297 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:01:39.289+0000 7f2ec3e7d540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:01:39.297 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:01:39.289+0000 7f2ec3e7d540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:01:39.297 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:01:39.289+0000 7f2ec3e7d540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:01:39.368 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:01:39.361+0000 7f2ec3e7d540 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-18T15:01:45.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-18T15:01:45.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:01:45.964 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T15:01:45.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:01:45.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:01:46.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:01:46.353 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T15:01:46.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-18T15:01:46.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:01:46.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:01:46.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:01:46.379 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:01:46.369+0000 7f70bd093540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:01:46.382 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:01:46.373+0000 7f70bd093540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:01:46.384 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:01:46.373+0000 7f70bd093540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:01:46.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:01:46.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:01:46.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:01:46.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:01:46.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:01:46.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:46.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:01:46.695 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:01:46.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:46.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:01:46.922 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:01:46.913+0000 7f70bd093540 -1 Falling back to public interface 2024-09-18T15:01:47.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:48.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:48.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:48.077 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:01:48.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:01:48.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:48.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:01:48.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:49.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:49.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:49.390 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:01:49.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:01:49.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:49.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:01:49.468 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:01:49.461+0000 7f70bd093540 -1 osd.1 0 log_to_monitors true 2024-09-18T15:01:49.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:50.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:50.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:50.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:01:50.713 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:01:50.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:50.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:01:51.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:52.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:52.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:52.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:01:52.259 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:01:52.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:52.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:01:52.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:53.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:53.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:53.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:01:53.575 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:01:53.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:53.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:01:54.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:55.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:55.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:55.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:01:55.075 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:01:55.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:55.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:01:55.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:56.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:56.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:56.507 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:01:56.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:01:56.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:56.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:01:56.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:57.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:57.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:57.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:01:57.824 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:01:57.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:57.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:01:58.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:01:59.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:01:59.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:01:59.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:01:59.294 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:01:59.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:01:59.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:01:59.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:00.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:00.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:00.605 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:02:00.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:02:00.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:00.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:02:00.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:01.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:01.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:01.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:02:01.922 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:02:01.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:01.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:02:02.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:03.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:03.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:03.264 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:02:03.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:02:03.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:03.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:02:03.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:04.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:04.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:04.605 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:02:04.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:02:04.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:04.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:02:05.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:06.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:06.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:06.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:02:06.088 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:02:06.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:06.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:02:06.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:07.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:07.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:07.481 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:02:07.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:02:07.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:07.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:02:07.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:08.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:08.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:08.889 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:02:08.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:02:08.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:08.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:02:09.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:10.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:10.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:10.382 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:02:10.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:02:10.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:10.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:02:10.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:11.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:11.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:11.810 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:02:11.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:02:11.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:11.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:02:12.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:13.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:13.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:13.246 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:02:13.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:02:13.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:13.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:02:13.701 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/186045,v1:127.0.0.1:6811/186045] [v2:127.0.0.1:6812/186045,v1:127.0.0.1:6813/186045] exists,up fe030fbb-d4c2-4e9c-b172-7695dc6417ed 2024-09-18T15:02:13.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:02:13.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:02:13.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:02:13.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:497: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-09-18T15:02:13.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-18T15:02:13.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:02:13.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:02:13.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T15:02:13.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:02:13.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:02:13.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:02:13.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:02:13.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:02:13.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:02:13.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T15:02:13.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T15:02:13.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:02:13.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:02:13.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:02:13.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:02:13.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:02:13.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:02:13.706 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:02:13.706 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:02:13.706 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:02:13.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:02:13.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:02:13.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:02:13.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:02:13.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:02:13.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:02:13.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:02:13.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:02:13.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:02:13.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:02:13.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:02:13.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:02:13.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-18T15:02:13.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-18T15:02:13.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:02:13.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d739e448-bc41-4c33-a073-16d33143757e 2024-09-18T15:02:13.710 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 d739e448-bc41-4c33-a073-16d33143757e 2024-09-18T15:02:13.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d739e448-bc41-4c33-a073-16d33143757e' 2024-09-18T15:02:13.710 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:02:13.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB16+pmy6xSKxAA+eW1YWsnw7Jf8z7KyJm0jw== 2024-09-18T15:02:13.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB16+pmy6xSKxAA+eW1YWsnw7Jf8z7KyJm0jw=="}' 2024-09-18T15:02:13.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d739e448-bc41-4c33-a073-16d33143757e -i td/osd-scrub-repair/2/new.json 2024-09-18T15:02:14.128 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:02:14.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-18T15:02:14.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQB16+pmy6xSKxAA+eW1YWsnw7Jf8z7KyJm0jw== --osd-uuid d739e448-bc41-4c33-a073-16d33143757e 2024-09-18T15:02:14.180 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:02:14.168+0000 7f0c51902540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:02:14.190 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:02:14.180+0000 7f0c51902540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:02:14.192 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:02:14.180+0000 7f0c51902540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:02:14.193 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:02:14.184+0000 7f0c51902540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T15:02:14.193 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:02:14.184+0000 7f0c51902540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T15:02:14.193 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:02:14.184+0000 7f0c51902540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T15:02:14.264 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:02:14.256+0000 7f0c51902540 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-18T15:02:20.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-18T15:02:20.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:02:20.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T15:02:20.883 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T15:02:20.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:02:21.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T15:02:21.271 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T15:02:21.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:02:21.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:02:21.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-18T15:02:21.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:02:21.303 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:02:21.292+0000 7f85afa39540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:02:21.306 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:02:21.296+0000 7f85afa39540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:02:21.307 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:02:21.296+0000 7f85afa39540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:02:21.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T15:02:21.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:02:21.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:02:21.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:02:21.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:02:21.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:21.567 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:02:21.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:02:21.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:21.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:02:21.611 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:02:21.600+0000 7f85afa39540 -1 Falling back to public interface 2024-09-18T15:02:21.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:22.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:22.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:22.842 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:02:22.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:02:22.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:22.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:02:23.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:24.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:24.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:24.156 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:02:24.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:02:24.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:24.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:02:24.304 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:02:24.292+0000 7f85afa39540 -1 osd.2 0 log_to_monitors true 2024-09-18T15:02:24.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:25.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:25.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:25.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:02:25.626 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:02:25.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:25.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:02:25.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:26.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:26.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:26.946 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:02:26.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:02:26.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:26.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:02:27.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:28.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:28.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:28.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:02:28.272 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:02:28.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:28.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:02:28.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:29.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:29.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:29.590 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:02:29.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:02:29.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:29.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:02:29.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:30.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:30.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:30.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:02:30.910 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:02:30.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:30.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:02:31.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:32.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:32.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:32.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:02:32.268 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:02:32.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:02:32.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:32.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:33.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:33.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:33.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:02:33.587 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:02:33.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:33.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:02:34.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:35.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:35.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:35.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:02:35.221 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:02:35.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:35.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:02:35.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:36.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:36.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:36.624 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:02:36.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:02:36.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:36.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:02:37.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:38.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:38.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:38.142 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:02:38.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:02:38.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:38.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:02:38.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:39.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:39.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:39.577 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:02:39.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:02:39.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:39.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:02:40.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:41.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:41.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:41.014 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:02:41.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:02:41.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:41.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:02:41.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:42.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:42.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:42.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:02:42.616 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:02:42.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:42.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:02:42.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:44.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:44.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:44.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:02:44.002 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:02:44.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:44.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:02:44.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:45.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:45.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:45.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:02:45.520 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:02:45.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:45.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:02:45.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:46.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:46.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:46.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:02:46.929 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:02:46.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:46.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:02:47.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:02:48.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:02:48.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:48.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:02:48.350 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:02:48.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:48.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:02:48.733 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/187849,v1:127.0.0.1:6819/187849] [v2:127.0.0.1:6820/187849,v1:127.0.0.1:6821/187849] exists,up d739e448-bc41-4c33-a073-16d33143757e 2024-09-18T15:02:48.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:02:48.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:02:48.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:02:48.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:501: TEST_auto_repair_bluestore_scrub: create_pool testpool 1 1 2024-09-18T15:02:48.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-09-18T15:02:49.178 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'testpool' created 2024-09-18T15:02:49.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:02:50.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:502: TEST_auto_repair_bluestore_scrub: ceph osd pool set testpool size 2 2024-09-18T15:02:50.730 INFO:tasks.workunit.client.0.smithi046.stderr:set pool 1 size to 2 2024-09-18T15:02:50.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:503: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-09-18T15:02:50.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:02:50.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:02:50.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:02:50.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:02:50.744 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:02:50.745 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:02:50.745 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:02:50.746 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:02:50.746 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:02:50.746 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:02:50.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:02:50.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:02:50.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:02:50.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:02:50.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:02:51.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:02:51.297 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:02:51.297 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:02:51.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:02:51.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:02:51.297 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:02:51.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836502 2024-09-18T15:02:51.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836502 2024-09-18T15:02:51.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836502' 2024-09-18T15:02:51.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:02:51.632 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:02:51.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672976 2024-09-18T15:02:51.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672976 2024-09-18T15:02:51.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836502 1-42949672976' 2024-09-18T15:02:51.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:02:51.784 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:02:51.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509444 2024-09-18T15:02:51.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509444 2024-09-18T15:02:51.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836502 1-42949672976 2-64424509444' 2024-09-18T15:02:51.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:02:51.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836502 2024-09-18T15:02:51.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:02:51.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:02:51.927 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836502 2024-09-18T15:02:51.927 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:02:51.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836502 2024-09-18T15:02:51.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-18T15:02:51.929 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836502 2024-09-18T15:02:51.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:02:52.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836501 -lt 21474836502 2024-09-18T15:02:52.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:02:53.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:02:53.324 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:02:53.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836502 -lt 21474836502 2024-09-18T15:02:53.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:02:53.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672976 2024-09-18T15:02:53.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:02:53.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:02:53.659 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672976 2024-09-18T15:02:53.659 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:02:53.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672976 2024-09-18T15:02:53.660 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672976 2024-09-18T15:02:53.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-18T15:02:53.661 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:02:54.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672976 -lt 42949672976 2024-09-18T15:02:54.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:02:54.007 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509444 2024-09-18T15:02:54.007 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:02:54.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:02:54.009 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509444 2024-09-18T15:02:54.009 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:02:54.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509444 2024-09-18T15:02:54.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-09-18T15:02:54.012 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509444 2024-09-18T15:02:54.012 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:02:54.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509444 -lt 64424509444 2024-09-18T15:02:54.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:02:54.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:02:54.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:02:54.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:02:54.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:02:54.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:02:54.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:02:54.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:02:54.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:02:54.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:02:54.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:02:55.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:02:55.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:02:55.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:02:55.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:02:55.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:02:55.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:02:55.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:02:55.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:506: TEST_auto_repair_bluestore_scrub: local payload=ABCDEF 2024-09-18T15:02:55.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:507: TEST_auto_repair_bluestore_scrub: echo ABCDEF 2024-09-18T15:02:55.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-18T15:02:55.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-09-18T15:02:55.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-09-18T15:02:55.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-09-18T15:02:55.709 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-09-18T15:02:55.710 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-09-18T15:02:55.710 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T15:02:55.710 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T15:02:55.710 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:02:56.025 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T15:02:56.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T15:02:56.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T15:02:56.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-18T15:02:56.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:02:56.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:02:56.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:02:56.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:02:56.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-18T15:02:56.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:02:56.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:02:56.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:02:56.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:02:56.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:02:56.384 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:02:56.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:02:56.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:02:56.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:02:56.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:02:56.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:02:56.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-18T15:02:56.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:02:56.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:02:56.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:02:56.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:02:56.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:02:56.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-18T15:02:58.325 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-18T15:02:58.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-18T15:02:58.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:02:58.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:02:58.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:02:58.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:02:58.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:02:58.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:02:58.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:02:58.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:02:58.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:02:58.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:02:58.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:02:58.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:02:58.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:02:58.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:02:58.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:02:58.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:02:58.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:02:58.867 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:02:58.867 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:02:58.867 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:02:58.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:02:58.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:02:58.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:02:58.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:02:58.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:02:58.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:02:58.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:02:58.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:02:58.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:02:58.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:02:58.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-18T15:02:58.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:02:58.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:02:58.870 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:02:58.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-18T15:02:58.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:02:58.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:02:58.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:02:58.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:02:58.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:02:58.907 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:02:58.896+0000 7fe6951cd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:02:58.910 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:02:58.900+0000 7fe6951cd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:02:58.912 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:02:58.900+0000 7fe6951cd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:02:59.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:02:59.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:02:59.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:02:59.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:02:59.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:02:59.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:02:59.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:02:59.190 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:02:59.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:02:59.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:02:59.447 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:02:59.436+0000 7fe6951cd540 -1 Falling back to public interface 2024-09-18T15:02:59.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:03:00.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:03:00.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:03:00.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:03:00.543 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:03:00.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:03:00.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:03:00.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:03:01.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:03:01.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:03:01.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:03:01.885 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:03:01.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:03:01.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:03:02.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:03:02.295 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:03:02.288+0000 7fe6951cd540 -1 osd.0 21 log_to_monitors true 2024-09-18T15:03:03.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:03:03.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:03:03.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:03:03.203 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:03:03.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:03:03.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:03:03.519 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/189722,v1:127.0.0.1:6803/189722] [v2:127.0.0.1:6804/189722,v1:127.0.0.1:6805/189722] exists,up 4ff80dfa-aa33-4bc0-8200-c0feb377d9ec 2024-09-18T15:03:03.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:03:03.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:03:03.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:03:03.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:03:03.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:03:03.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:03:03.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:03:03.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:03:03.520 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:03:03.521 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:03:03.521 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:03:03.521 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:03:03.521 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:03:03.521 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:03:03.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:03:03.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:03:03.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:03:03.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:03:03.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:03:04.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:03:04.175 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:03:04.175 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:03:04.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:03:04.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:03:04.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:03:04.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182402 2024-09-18T15:03:04.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182402 2024-09-18T15:03:04.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182402' 2024-09-18T15:03:04.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:03:04.311 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:03:04.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672980 2024-09-18T15:03:04.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672980 2024-09-18T15:03:04.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182402 1-42949672980' 2024-09-18T15:03:04.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:03:04.450 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:03:04.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509448 2024-09-18T15:03:04.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509448 2024-09-18T15:03:04.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182402 1-42949672980 2-64424509448' 2024-09-18T15:03:04.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:03:04.590 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-107374182402 2024-09-18T15:03:04.590 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:03:04.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:03:04.593 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-107374182402 2024-09-18T15:03:04.594 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:03:04.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182402 2024-09-18T15:03:04.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-09-18T15:03:04.595 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 107374182402 2024-09-18T15:03:04.596 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:03:04.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 107374182402 2024-09-18T15:03:04.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:03:05.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:03:05.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:03:06.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182402 -lt 107374182402 2024-09-18T15:03:06.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:03:06.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672980 2024-09-18T15:03:06.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:03:06.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:03:06.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672980 2024-09-18T15:03:06.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:03:06.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672980 2024-09-18T15:03:06.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-09-18T15:03:06.219 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 42949672980 2024-09-18T15:03:06.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:03:06.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672980 -lt 42949672980 2024-09-18T15:03:06.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:03:06.562 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509448 2024-09-18T15:03:06.562 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:03:06.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:03:06.565 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509448 2024-09-18T15:03:06.565 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:03:06.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509448 2024-09-18T15:03:06.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-18T15:03:06.567 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509448 2024-09-18T15:03:06.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:03:06.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509448 -lt 64424509448 2024-09-18T15:03:06.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:03:06.916 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:06.916 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:07.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:03:07.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:07.293 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:07.293 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:07.293 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:07.293 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:07.293 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:07.294 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:07.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:03:07.637 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:07.637 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:07.638 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:08.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:03:08.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:03:08.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:03:08.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:514: TEST_auto_repair_bluestore_scrub: get_pg testpool SOMETHING 2024-09-18T15:03:08.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-09-18T15:03:08.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T15:03:08.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-09-18T15:03:08.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:03:08.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:514: TEST_auto_repair_bluestore_scrub: local pgid=1.0 2024-09-18T15:03:08.344 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:515: TEST_auto_repair_bluestore_scrub: get_primary testpool SOMETHING 2024-09-18T15:03:08.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-09-18T15:03:08.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T15:03:08.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T15:03:08.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:03:08.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:515: TEST_auto_repair_bluestore_scrub: local primary=1 2024-09-18T15:03:08.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:516: TEST_auto_repair_bluestore_scrub: get_last_scrub_stamp 1.0 2024-09-18T15:03:08.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:03:08.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:03:08.699 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:03:08.699 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:03:09.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:516: TEST_auto_repair_bluestore_scrub: local last_scrub_stamp=2024-09-18T15:02:49.086108+0000 2024-09-18T15:03:09.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:517: TEST_auto_repair_bluestore_scrub: ceph tell 1.0 scrub 2024-09-18T15:03:09.141 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T15:03:09.141 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T15:03:09.141 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T15:01:29.136527+0000" 2024-09-18T15:03:09.141 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T15:03:09.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:520: TEST_auto_repair_bluestore_scrub: wait_for_scrub 1.0 2024-09-18T15:02:49.086108+0000 2024-09-18T15:03:09.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-18T15:03:09.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:02:49.086108+0000 2024-09-18T15:03:09.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:03:09.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:03:09.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:03:09.152 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T15:03:09.152 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:03:09.152 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:03:09.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:03:09.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:03:09.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:02:49.086108+0000 '>' 2024-09-18T15:02:49.086108+0000 2024-09-18T15:03:09.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:03:10.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:03:10.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:03:10.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T15:03:10.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:03:10.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:03:10.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:03:10.452 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:03:10.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:02:49.086108+0000 '>' 2024-09-18T15:02:49.086108+0000 2024-09-18T15:03:10.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:03:11.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:03:11.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:03:11.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T15:03:11.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:03:11.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:03:11.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:03:11.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:03:12.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:02:49.086108+0000 '>' 2024-09-18T15:02:49.086108+0000 2024-09-18T15:03:12.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:03:13.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:03:13.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:03:13.090 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T15:03:13.090 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:03:13.090 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:03:13.091 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:03:13.091 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:03:13.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:02:49.086108+0000 '>' 2024-09-18T15:02:49.086108+0000 2024-09-18T15:03:13.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:03:14.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:03:14.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:03:14.415 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T15:03:14.415 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:03:14.415 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:03:14.415 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:03:14.415 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:03:14.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:02:49.086108+0000 '>' 2024-09-18T15:02:49.086108+0000 2024-09-18T15:03:14.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:03:15.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:03:15.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:03:15.720 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T15:03:15.720 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:03:15.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:03:15.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:03:15.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:03:16.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:03:10.614374+0000 '>' 2024-09-18T15:02:49.086108+0000 2024-09-18T15:03:16.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:03:16.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:521: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-09-18T15:03:16.319 INFO:tasks.workunit.client.0.smithi046.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T15:03:16.319 INFO:tasks.workunit.client.0.smithi046.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-09-18T15:03:10.959192+0000 21'1 26:71 [1,0] 1 [1,0] 1 21'1 2024-09-18T15:03:10.614374+0000 21'1 2024-09-18T15:03:10.614374+0000 0 1 periodic scrub scheduled @ 2024-09-19T15:03:10.614374+0000 1 0 2024-09-18T15:03:16.320 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-09-18T15:03:16.320 INFO:tasks.workunit.client.0.smithi046.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T15:03:16.323 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T15:03:16.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:523: TEST_auto_repair_bluestore_scrub: sleep 5 2024-09-18T15:03:21.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:524: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-09-18T15:03:21.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:03:21.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:03:21.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:03:21.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:03:21.335 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:03:21.336 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:03:21.336 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:03:21.336 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:03:21.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:03:21.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:03:21.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:03:21.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:03:21.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:03:21.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:03:21.571 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:03:21.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:03:21.878 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:03:21.878 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:03:21.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:03:21.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:03:21.879 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:03:22.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182406 2024-09-18T15:03:22.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182406 2024-09-18T15:03:22.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182406' 2024-09-18T15:03:22.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:03:22.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:03:22.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672984 2024-09-18T15:03:22.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672984 2024-09-18T15:03:22.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182406 1-42949672984' 2024-09-18T15:03:22.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:03:22.192 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:03:22.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509452 2024-09-18T15:03:22.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509452 2024-09-18T15:03:22.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182406 1-42949672984 2-64424509452' 2024-09-18T15:03:22.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:03:22.334 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-107374182406 2024-09-18T15:03:22.334 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:03:22.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:03:22.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-107374182406 2024-09-18T15:03:22.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:03:22.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182406 2024-09-18T15:03:22.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 107374182406' 2024-09-18T15:03:22.339 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 107374182406 2024-09-18T15:03:22.339 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:03:22.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182405 -lt 107374182406 2024-09-18T15:03:22.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:03:23.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:03:23.653 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:03:23.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182407 -lt 107374182406 2024-09-18T15:03:23.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:03:23.975 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672984 2024-09-18T15:03:23.975 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:03:23.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:03:23.977 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672984 2024-09-18T15:03:23.977 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:03:23.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672984 2024-09-18T15:03:23.979 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672984 2024-09-18T15:03:23.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-09-18T15:03:23.979 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:03:24.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672984 -lt 42949672984 2024-09-18T15:03:24.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:03:24.321 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509452 2024-09-18T15:03:24.322 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:03:24.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:03:24.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509452 2024-09-18T15:03:24.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:03:24.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509452 2024-09-18T15:03:24.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-18T15:03:24.327 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509452 2024-09-18T15:03:24.327 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:03:24.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509452 -lt 64424509452 2024-09-18T15:03:24.657 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:03:24.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:24.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:25.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:03:25.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:25.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:25.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:25.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:25.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:25.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:25.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:25.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:03:25.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:25.415 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:25.415 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:25.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:03:25.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:03:25.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:03:25.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:525: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-09-18T15:03:26.113 INFO:tasks.workunit.client.0.smithi046.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T15:03:26.113 INFO:tasks.workunit.client.0.smithi046.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-09-18T15:03:10.959192+0000 21'1 26:71 [1,0] 1 [1,0] 1 21'1 2024-09-18T15:03:10.614374+0000 21'1 2024-09-18T15:03:10.614374+0000 0 1 periodic scrub scheduled @ 2024-09-19T15:03:10.614374+0000 1 0 2024-09-18T15:03:26.113 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-09-18T15:03:26.113 INFO:tasks.workunit.client.0.smithi046.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T15:03:26.116 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T15:03:26.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-09-18T15:03:26.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-09-18T15:03:26.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-09-18T15:03:26.127 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-09-18T15:03:26.127 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-09-18T15:03:26.127 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T15:03:26.128 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T15:03:26.128 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:03:26.454 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T15:03:26.454 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T15:03:26.454 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T15:03:26.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-18T15:03:26.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:03:26.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:03:26.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:03:26.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:03:26.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-18T15:03:26.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:03:26.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:03:26.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:03:26.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:03:26.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:03:26.798 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:03:26.798 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:03:26.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:03:26.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:03:26.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:03:28.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:03:28.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-18T15:03:28.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:03:28.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:03:28.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:03:28.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:03:28.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:03:28.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-09-18T15:03:29.438 INFO:tasks.workunit.client.0.smithi046.stdout:_ 2024-09-18T15:03:29.439 INFO:tasks.workunit.client.0.smithi046.stdout:snapset 2024-09-18T15:03:29.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-18T15:03:29.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:03:29.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:03:29.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:03:29.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:03:29.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:03:29.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:03:29.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:03:29.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:03:29.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:03:29.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:03:29.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:03:29.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:03:29.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:03:29.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:03:29.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:03:29.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:03:29.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:03:29.987 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:03:29.987 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:03:29.987 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:03:29.987 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:03:29.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:03:29.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:03:29.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:03:29.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:03:29.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:03:29.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:03:29.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:03:29.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:03:29.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:03:29.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-18T15:03:29.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:03:29.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:03:29.988 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:03:29.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-18T15:03:29.990 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:03:29.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:03:29.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:03:29.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:03:29.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:03:30.023 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:03:30.015+0000 7f824bf03540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:03:30.026 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:03:30.019+0000 7f824bf03540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:03:30.028 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:03:30.023+0000 7f824bf03540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:03:30.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:03:30.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:03:30.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:03:30.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:03:30.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:03:30.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:03:30.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:03:30.310 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:03:30.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:03:30.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:03:30.559 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:03:30.551+0000 7f824bf03540 -1 Falling back to public interface 2024-09-18T15:03:30.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:03:31.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:03:31.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:03:31.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:03:31.666 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:03:31.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:03:31.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:03:31.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:03:32.959 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:03:32.951+0000 7f824bf03540 -1 osd.0 26 log_to_monitors true 2024-09-18T15:03:32.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:03:32.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:03:32.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:03:32.975 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:03:32.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:03:32.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:03:33.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:03:34.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:03:34.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:03:34.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:03:34.320 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:03:34.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:03:34.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:03:34.632 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/191809,v1:127.0.0.1:6803/191809] [v2:127.0.0.1:6804/191809,v1:127.0.0.1:6805/191809] exists,up 4ff80dfa-aa33-4bc0-8200-c0feb377d9ec 2024-09-18T15:03:34.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:03:34.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:03:34.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:03:34.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:03:34.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:03:34.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:03:34.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:03:34.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:03:34.634 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:03:34.634 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:03:34.634 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:03:34.635 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:03:34.635 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:03:34.635 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:03:34.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:03:34.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:03:34.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:03:34.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:03:34.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:03:35.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:03:35.235 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:03:35.235 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:03:35.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:03:35.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:03:35.236 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:03:35.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018882 2024-09-18T15:03:35.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018882 2024-09-18T15:03:35.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-128849018882' 2024-09-18T15:03:35.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:03:35.417 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:03:35.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672988 2024-09-18T15:03:35.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672988 2024-09-18T15:03:35.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-128849018882 1-42949672988' 2024-09-18T15:03:35.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:03:35.556 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:03:35.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509456 2024-09-18T15:03:35.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509456 2024-09-18T15:03:35.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-128849018882 1-42949672988 2-64424509456' 2024-09-18T15:03:35.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:03:35.728 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-128849018882 2024-09-18T15:03:35.728 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:03:35.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:03:35.731 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-128849018882 2024-09-18T15:03:35.732 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:03:35.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018882 2024-09-18T15:03:35.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-09-18T15:03:35.733 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 128849018882 2024-09-18T15:03:35.734 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:03:36.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 128849018882 2024-09-18T15:03:36.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:03:37.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:03:37.039 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:03:37.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018882 -lt 128849018882 2024-09-18T15:03:37.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:03:37.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672988 2024-09-18T15:03:37.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:03:37.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:03:37.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672988 2024-09-18T15:03:37.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:03:37.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672988 2024-09-18T15:03:37.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-09-18T15:03:37.363 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 42949672988 2024-09-18T15:03:37.364 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:03:37.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672988 -lt 42949672988 2024-09-18T15:03:37.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:03:37.702 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509456 2024-09-18T15:03:37.703 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:03:37.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:03:37.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509456 2024-09-18T15:03:37.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:03:37.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509456 2024-09-18T15:03:37.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-09-18T15:03:37.709 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509456 2024-09-18T15:03:37.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:03:38.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509456 -lt 64424509456 2024-09-18T15:03:38.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:03:38.015 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:38.015 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:38.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:03:38.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:38.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:38.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:38.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:38.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:38.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:38.436 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:38.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:03:38.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:38.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:38.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:39.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:03:39.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:03:39.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:03:39.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_scrub: rados --pool testpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-18T15:03:39.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:530: TEST_auto_repair_bluestore_scrub: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-18T15:03:39.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:531: TEST_auto_repair_bluestore_scrub: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-18T15:03:39.183 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T15:03:09.648+0000 7f70a0da6700 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] planned DEEP_SCRUB_ON_ERROR] scrubber : scrub_finish before flags: DEEP_SCRUB_ON_ERROR. repair state: no-repair. deep_scrub_on_error: 1 2024-09-18T15:03:39.183 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T15:03:09.648+0000 7f70a0da6700 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] ] scrubber : scrub_finish Try to auto repair after scrub errors 2024-09-18T15:03:39.183 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T15:03:09.648+0000 7f70a0da6700 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: ] ] scrubber : _scrub_finish info stats: valid 2024-09-18T15:03:39.184 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T15:03:10.612+0000 7f70a0da6700 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber : scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-18T15:03:39.184 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T15:03:10.612+0000 7f70a0da6700 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] ] scrubber : _scrub_finish info stats: valid 2024-09-18T15:03:39.184 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T15:03:10.612+0000 7f70a0da6700 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber : scrub_finish All may be fixed 2024-09-18T15:03:39.184 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: ceph pg 1.0 query 2024-09-18T15:03:39.184 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: jq .info.stats.stat_sum.num_objects_repaired 2024-09-18T15:03:39.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: COUNT=1 2024-09-18T15:03:39.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:535: TEST_auto_repair_bluestore_scrub: test 1 = 1 2024-09-18T15:03:39.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T15:03:39.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:03:39.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:03:39.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:03:39.320 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:03:39.321 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:03:39.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:03:39.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:03:39.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:03:39.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:03:39.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:03:39.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:03:39.443 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:03:39.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:03:39.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:03:39.446 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:03:39.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:03:39.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:03:39.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:03:39.449 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:03:39.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:03:39.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:03:39.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:03:39.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:03:39.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:03:39.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:03:39.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:03:39.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:03:39.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:03:39.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T15:03:39.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T15:03:39.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T15:03:39.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T15:03:39.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:03:39.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:03:39.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:03:39.477 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:03:39.477 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:03:39.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:03:39.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:03:39.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:03:39.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:03:39.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:03:39.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:03:39.484 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:03:39.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:03:39.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:03:39.487 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:03:39.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:03:39.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:03:39.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:03:39.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:03:39.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:03:39.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:03:39.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:03:39.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:03:39.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:03:39.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:03:39.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:03:39.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:03:39.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:03:39.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T15:03:39.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:03:39.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:03:39.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:03:39.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T15:03:39.503 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:03:39.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:03:39.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:03:39.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T15:03:39.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_auto_repair_bluestore_tag td/osd-scrub-repair 2024-09-18T15:03:39.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:387: TEST_auto_repair_bluestore_tag: local dir=td/osd-scrub-repair 2024-09-18T15:03:39.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:388: TEST_auto_repair_bluestore_tag: local poolname=testpool 2024-09-18T15:03:39.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:391: TEST_auto_repair_bluestore_tag: run_mon td/osd-scrub-repair a 2024-09-18T15:03:39.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T15:03:39.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:03:39.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:03:39.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:03:39.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T15:03:39.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-18T15:03:39.925 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:03:39.925 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:03:39.925 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:03:39.926 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:03:39.926 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:03:39.926 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:03:39.926 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:03:39.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T15:03:40.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:03:40.215 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:03:40.215 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:03:40.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:03:40.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:03:40.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:03:40.217 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:03:40.217 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:03:40.217 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:03:40.218 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:03:40.218 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:03:40.218 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:03:40.219 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:03:40.219 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:03:40.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T15:03:40.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:03:40.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:03:40.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:03:40.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:03:40.310 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:03:40.310 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:03:40.310 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:03:40.310 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:03:40.311 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:03:40.311 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:03:40.311 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:03:40.312 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:03:40.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:03:40.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T15:03:40.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:392: TEST_auto_repair_bluestore_tag: run_mgr td/osd-scrub-repair x 2024-09-18T15:03:40.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T15:03:40.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:03:40.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:03:40.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:03:40.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T15:03:40.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:03:41.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:03:41.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:03:41.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:03:41.249 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:03:41.249 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:03:41.249 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:03:41.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:03:41.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:03:41.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:03:41.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: TEST_auto_repair_bluestore_tag: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-18T15:03:41.287 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:400: TEST_auto_repair_bluestore_tag: seq 0 2 2024-09-18T15:03:41.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:400: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-09-18T15:03:41.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-18T15:03:41.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:03:41.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:03:41.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:03:41.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:03:41.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:03:41.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:03:41.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:03:41.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:03:41.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:03:41.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:03:41.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:03:41.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:03:41.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:03:41.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:03:41.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:03:41.292 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:03:41.292 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:03:41.294 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:03:41.294 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:03:41.294 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:03:41.294 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:03:41.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:03:41.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:03:41.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:03:41.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:03:41.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:03:41.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:03:41.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:03:41.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:03:41.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:03:41.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:03:41.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:03:41.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-18T15:03:41.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:03:41.295 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:03:41.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f917995b-d31d-42dd-ab17-1bf7b28590db 2024-09-18T15:03:41.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f917995b-d31d-42dd-ab17-1bf7b28590db' 2024-09-18T15:03:41.296 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 f917995b-d31d-42dd-ab17-1bf7b28590db 2024-09-18T15:03:41.296 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:03:41.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDN6+pm8JfOEhAAgfyxoVbC1Lx1c/zDo3W9XQ== 2024-09-18T15:03:41.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDN6+pm8JfOEhAAgfyxoVbC1Lx1c/zDo3W9XQ=="}' 2024-09-18T15:03:41.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f917995b-d31d-42dd-ab17-1bf7b28590db -i td/osd-scrub-repair/0/new.json 2024-09-18T15:03:41.666 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:03:41.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-18T15:03:41.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQDN6+pm8JfOEhAAgfyxoVbC1Lx1c/zDo3W9XQ== --osd-uuid f917995b-d31d-42dd-ab17-1bf7b28590db 2024-09-18T15:03:41.718 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:03:41.711+0000 7fce4cf48540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:03:41.721 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:03:41.715+0000 7fce4cf48540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:03:41.723 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:03:41.715+0000 7fce4cf48540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:03:41.724 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:03:41.719+0000 7fce4cf48540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:03:41.724 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:03:41.719+0000 7fce4cf48540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:03:41.725 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:03:41.719+0000 7fce4cf48540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:03:41.823 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:03:41.815+0000 7fce4cf48540 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-18T15:03:48.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T15:03:48.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:03:48.112 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T15:03:48.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:03:48.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:03:48.503 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T15:03:48.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:03:48.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-18T15:03:48.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:03:48.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:03:48.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:03:48.535 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:03:48.527+0000 7f28a60ce540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:03:48.537 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:03:48.531+0000 7f28a60ce540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:03:48.539 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:03:48.531+0000 7f28a60ce540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:03:48.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:03:48.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:03:48.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:03:48.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:03:48.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:03:48.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:03:48.809 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:03:48.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:03:48.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:03:48.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:03:49.075 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:03:49.067+0000 7f28a60ce540 -1 Falling back to public interface 2024-09-18T15:03:49.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:03:50.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:03:50.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:03:50.107 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:03:50.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:03:50.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:03:50.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:03:50.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:03:51.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:03:51.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:03:51.420 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:03:51.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:03:51.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:03:51.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:03:51.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:03:51.851 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:03:51.843+0000 7f28a60ce540 -1 osd.0 0 log_to_monitors true 2024-09-18T15:03:52.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:03:52.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:03:52.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:03:52.737 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:03:52.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:03:52.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:03:53.052 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/193947,v1:127.0.0.1:6803/193947] [v2:127.0.0.1:6804/193947,v1:127.0.0.1:6805/193947] exists,up f917995b-d31d-42dd-ab17-1bf7b28590db 2024-09-18T15:03:53.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:03:53.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:03:53.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:03:53.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:400: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-09-18T15:03:53.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-18T15:03:53.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:03:53.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:03:53.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:03:53.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:03:53.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:03:53.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:03:53.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:03:53.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:03:53.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:03:53.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:03:53.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:03:53.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:03:53.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:03:53.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:03:53.056 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:03:53.056 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:03:53.056 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:03:53.056 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:03:53.056 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:03:53.056 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:03:53.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:03:53.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:03:53.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:03:53.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:03:53.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:03:53.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:03:53.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:03:53.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:03:53.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:03:53.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:03:53.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:03:53.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:03:53.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-18T15:03:53.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:03:53.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:03:53.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b857af42-416a-4265-920c-755513a95b3c 2024-09-18T15:03:53.062 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 b857af42-416a-4265-920c-755513a95b3c 2024-09-18T15:03:53.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b857af42-416a-4265-920c-755513a95b3c' 2024-09-18T15:03:53.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:03:53.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDZ6+pmxAvFBBAAV60r3VVCS0EsTdyu1nEvsw== 2024-09-18T15:03:53.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDZ6+pmxAvFBBAAV60r3VVCS0EsTdyu1nEvsw=="}' 2024-09-18T15:03:53.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b857af42-416a-4265-920c-755513a95b3c -i td/osd-scrub-repair/1/new.json 2024-09-18T15:03:53.468 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:03:53.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-18T15:03:53.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQDZ6+pmxAvFBBAAV60r3VVCS0EsTdyu1nEvsw== --osd-uuid b857af42-416a-4265-920c-755513a95b3c 2024-09-18T15:03:53.517 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:03:53.511+0000 7f7a2862f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:03:53.527 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:03:53.519+0000 7f7a2862f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:03:53.529 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:03:53.523+0000 7f7a2862f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:03:53.530 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:03:53.523+0000 7f7a2862f540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:03:53.530 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:03:53.523+0000 7f7a2862f540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:03:53.530 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:03:53.523+0000 7f7a2862f540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:03:53.600 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:03:53.591+0000 7f7a2862f540 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-18T15:04:00.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-18T15:04:00.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:04:00.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:04:00.420 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T15:04:00.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:04:00.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:04:00.813 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T15:04:00.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:04:00.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-18T15:04:00.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:04:00.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:04:00.845 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:00.839+0000 7fe4f8781540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:04:00.849 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:00.843+0000 7fe4f8781540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:04:00.851 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:00.843+0000 7fe4f8781540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:04:01.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:04:01.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:04:01.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:04:01.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:04:01.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:04:01.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:04:01.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:04:01.132 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:04:01.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:04:01.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:04:01.407 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:01.399+0000 7fe4f8781540 -1 Falling back to public interface 2024-09-18T15:04:01.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:04:02.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:04:02.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:04:02.476 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:04:02.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:04:02.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:04:02.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:04:02.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:04:03.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:04:03.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:04:03.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:04:03.790 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:04:03.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:04:03.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:04:04.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:04:04.224 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:04.215+0000 7fe4f8781540 -1 osd.1 0 log_to_monitors true 2024-09-18T15:04:05.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:04:05.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:04:05.099 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:04:05.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:04:05.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:04:05.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:04:05.420 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/195346,v1:127.0.0.1:6811/195346] [v2:127.0.0.1:6812/195346,v1:127.0.0.1:6813/195346] exists,up b857af42-416a-4265-920c-755513a95b3c 2024-09-18T15:04:05.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:04:05.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:04:05.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:04:05.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:400: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-09-18T15:04:05.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-18T15:04:05.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:04:05.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:04:05.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T15:04:05.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:04:05.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:04:05.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:04:05.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:04:05.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:04:05.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:04:05.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T15:04:05.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T15:04:05.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:04:05.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:04:05.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:04:05.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:04:05.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:04:05.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:04:05.425 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:04:05.425 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:04:05.425 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:04:05.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:04:05.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:04:05.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:04:05.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:04:05.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:04:05.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:04:05.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:04:05.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:04:05.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:04:05.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:04:05.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:04:05.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:04:05.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-18T15:04:05.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-18T15:04:05.426 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:04:05.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1ba1e0ad-189c-4d21-a04e-21d469ee4571 2024-09-18T15:04:05.427 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 1ba1e0ad-189c-4d21-a04e-21d469ee4571 2024-09-18T15:04:05.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 1ba1e0ad-189c-4d21-a04e-21d469ee4571' 2024-09-18T15:04:05.427 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:04:05.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDl6+pma4wTGhAAAm5OcDVvuMGieLTmNfzuCg== 2024-09-18T15:04:05.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDl6+pma4wTGhAAAm5OcDVvuMGieLTmNfzuCg=="}' 2024-09-18T15:04:05.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1ba1e0ad-189c-4d21-a04e-21d469ee4571 -i td/osd-scrub-repair/2/new.json 2024-09-18T15:04:05.899 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:04:05.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-18T15:04:05.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQDl6+pma4wTGhAAAm5OcDVvuMGieLTmNfzuCg== --osd-uuid 1ba1e0ad-189c-4d21-a04e-21d469ee4571 2024-09-18T15:04:05.957 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:05.951+0000 7ff82bb1f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:04:05.960 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:05.951+0000 7ff82bb1f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:04:05.961 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:05.955+0000 7ff82bb1f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:04:05.963 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:05.955+0000 7ff82bb1f540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T15:04:05.963 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:05.955+0000 7ff82bb1f540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T15:04:05.963 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:05.955+0000 7ff82bb1f540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T15:04:06.028 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:06.019+0000 7ff82bb1f540 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-18T15:04:12.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-18T15:04:12.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:04:12.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T15:04:12.853 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T15:04:12.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:04:13.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T15:04:13.241 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T15:04:13.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-18T15:04:13.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:04:13.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:04:13.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:04:13.274 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:13.267+0000 7fd492d1d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:04:13.276 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:13.271+0000 7fd492d1d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:04:13.278 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:13.271+0000 7fd492d1d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:04:13.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T15:04:13.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:04:13.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:04:13.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:04:13.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:04:13.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:04:13.609 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:04:13.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:04:13.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:04:13.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:04:13.842 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:13.835+0000 7fd492d1d540 -1 Falling back to public interface 2024-09-18T15:04:13.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:04:14.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:04:14.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:04:14.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:04:14.918 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:04:14.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:04:14.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:04:15.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:04:16.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:04:16.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:04:16.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:04:16.227 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:04:16.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:04:16.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:04:16.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:04:16.600 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:16.591+0000 7fd492d1d540 -1 osd.2 0 log_to_monitors true 2024-09-18T15:04:17.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:04:17.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:04:17.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:04:17.542 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:04:17.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:04:17.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:04:17.898 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/196745,v1:127.0.0.1:6819/196745] [v2:127.0.0.1:6820/196745,v1:127.0.0.1:6821/196745] exists,up 1ba1e0ad-189c-4d21-a04e-21d469ee4571 2024-09-18T15:04:17.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:04:17.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:04:17.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:04:17.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:404: TEST_auto_repair_bluestore_tag: create_pool testpool 1 1 2024-09-18T15:04:17.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-09-18T15:04:18.386 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'testpool' created 2024-09-18T15:04:18.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:04:19.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:405: TEST_auto_repair_bluestore_tag: ceph osd pool set testpool size 2 2024-09-18T15:04:19.841 INFO:tasks.workunit.client.0.smithi046.stderr:set pool 1 size to 2 2024-09-18T15:04:19.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: TEST_auto_repair_bluestore_tag: wait_for_clean 2024-09-18T15:04:19.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:04:19.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:04:19.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:04:19.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:04:19.854 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:04:19.854 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:04:19.854 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:04:19.855 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:04:19.855 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:04:19.855 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:04:20.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:04:20.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:04:20.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:04:20.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:04:20.079 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:04:20.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:04:20.407 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:04:20.407 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:04:20.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:04:20.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:04:20.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:04:20.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836487 2024-09-18T15:04:20.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836487 2024-09-18T15:04:20.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836487' 2024-09-18T15:04:20.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:04:20.566 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:04:20.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672966 2024-09-18T15:04:20.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672966 2024-09-18T15:04:20.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836487 1-42949672966' 2024-09-18T15:04:20.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:04:20.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:04:20.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509443 2024-09-18T15:04:20.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509443 2024-09-18T15:04:20.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836487 1-42949672966 2-64424509443' 2024-09-18T15:04:20.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:04:20.921 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836487 2024-09-18T15:04:20.921 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:04:20.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:04:20.925 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836487 2024-09-18T15:04:20.925 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:04:20.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836487 2024-09-18T15:04:20.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-09-18T15:04:20.927 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836487 2024-09-18T15:04:20.928 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:04:21.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836487 -lt 21474836487 2024-09-18T15:04:21.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:04:21.238 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672966 2024-09-18T15:04:21.239 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:04:21.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:04:21.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672966 2024-09-18T15:04:21.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:04:21.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672966 2024-09-18T15:04:21.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-09-18T15:04:21.251 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672966 2024-09-18T15:04:21.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:04:21.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672964 -lt 42949672966 2024-09-18T15:04:21.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:04:22.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:04:22.589 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:04:22.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672966 -lt 42949672966 2024-09-18T15:04:22.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:04:22.900 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509443 2024-09-18T15:04:22.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:04:22.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:04:22.904 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509443 2024-09-18T15:04:22.904 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:04:22.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509443 2024-09-18T15:04:22.906 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509443 2024-09-18T15:04:22.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2024-09-18T15:04:22.907 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:04:23.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509443 -lt 64424509443 2024-09-18T15:04:23.278 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:04:23.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:23.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:23.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:04:23.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:23.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:23.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:23.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:23.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:23.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:23.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:24.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:04:24.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:24.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:24.055 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:24.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:04:24.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:04:24.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:04:24.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: TEST_auto_repair_bluestore_tag: local payload=ABCDEF 2024-09-18T15:04:24.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:410: TEST_auto_repair_bluestore_tag: echo ABCDEF 2024-09-18T15:04:24.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-18T15:04:24.589 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:415: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-09-18T15:04:24.589 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-09-18T15:04:24.589 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-09-18T15:04:24.590 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-09-18T15:04:24.590 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-09-18T15:04:24.590 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T15:04:24.591 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T15:04:24.591 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:04:24.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T15:04:24.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T15:04:24.902 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T15:04:25.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:415: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-18T15:04:25.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:04:25.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:04:25.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:04:25.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:04:25.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-18T15:04:25.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:04:25.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:04:25.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:04:25.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:04:25.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:04:25.195 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:04:25.195 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:04:25.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:04:25.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:04:25.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:04:25.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:04:25.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-18T15:04:25.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:04:25.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:04:25.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:04:25.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:04:25.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:04:25.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-18T15:04:26.989 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-18T15:04:27.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-18T15:04:27.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:04:27.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:04:27.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:04:27.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:04:27.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:04:27.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:04:27.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:04:27.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:04:27.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:04:27.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:04:27.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:04:27.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:04:27.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:04:27.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:04:27.540 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:04:27.540 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:04:27.540 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:04:27.540 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:04:27.540 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:04:27.540 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:04:27.541 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:04:27.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:04:27.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:04:27.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:04:27.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:04:27.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:04:27.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:04:27.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:04:27.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:04:27.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:04:27.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-18T15:04:27.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:04:27.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:04:27.542 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:04:27.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-18T15:04:27.542 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:04:27.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:04:27.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:04:27.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:04:27.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:04:27.576 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:27.566+0000 7f2c8625b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:04:27.579 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:27.570+0000 7f2c8625b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:04:27.581 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:27.574+0000 7f2c8625b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:04:27.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:04:27.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:04:27.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:04:27.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:04:27.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:04:27.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:04:27.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:04:27.858 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:04:27.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:04:27.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:04:28.123 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:28.114+0000 7f2c8625b540 -1 Falling back to public interface 2024-09-18T15:04:28.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:04:29.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:04:29.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:04:29.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:04:29.156 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:04:29.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:04:29.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:04:29.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:04:30.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:04:30.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:04:30.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:04:30.477 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:04:30.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:04:30.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:04:30.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:04:30.842 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:30.834+0000 7f2c8625b540 -1 osd.0 21 log_to_monitors true 2024-09-18T15:04:31.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:04:31.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:04:31.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:04:31.806 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:04:31.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:04:31.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:04:32.180 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/198214,v1:127.0.0.1:6803/198214] [v2:127.0.0.1:6804/198214,v1:127.0.0.1:6805/198214] exists,up f917995b-d31d-42dd-ab17-1bf7b28590db 2024-09-18T15:04:32.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:04:32.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:04:32.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:04:32.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:04:32.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:04:32.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:04:32.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:04:32.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:04:32.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:04:32.182 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:04:32.182 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:04:32.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:04:32.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:04:32.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:04:32.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:04:32.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:04:32.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:04:32.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:04:32.389 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:04:32.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:04:32.723 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:04:32.724 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:04:32.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:04:32.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:04:32.724 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:04:32.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182402 2024-09-18T15:04:32.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182402 2024-09-18T15:04:32.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182402' 2024-09-18T15:04:32.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:04:32.954 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:04:33.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672969 2024-09-18T15:04:33.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672969 2024-09-18T15:04:33.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182402 1-42949672969' 2024-09-18T15:04:33.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:04:33.099 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:04:33.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509447 2024-09-18T15:04:33.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509447 2024-09-18T15:04:33.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182402 1-42949672969 2-64424509447' 2024-09-18T15:04:33.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:04:33.238 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-107374182402 2024-09-18T15:04:33.238 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:04:33.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:04:33.242 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-107374182402 2024-09-18T15:04:33.242 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:04:33.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182402 2024-09-18T15:04:33.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-09-18T15:04:33.244 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 107374182402 2024-09-18T15:04:33.244 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:04:33.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 107374182402 2024-09-18T15:04:33.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:04:34.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:04:34.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:04:34.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182402 -lt 107374182402 2024-09-18T15:04:34.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:04:34.912 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672969 2024-09-18T15:04:34.912 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:04:34.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:04:34.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672969 2024-09-18T15:04:34.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:04:34.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672969 2024-09-18T15:04:34.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-09-18T15:04:34.917 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 42949672969 2024-09-18T15:04:34.918 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:04:35.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672969 -lt 42949672969 2024-09-18T15:04:35.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:04:35.257 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509447 2024-09-18T15:04:35.257 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:04:35.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:04:35.259 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509447 2024-09-18T15:04:35.260 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:04:35.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509447 2024-09-18T15:04:35.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-09-18T15:04:35.262 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509447 2024-09-18T15:04:35.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:04:35.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509447 -lt 64424509447 2024-09-18T15:04:35.608 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:04:35.608 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:35.609 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:35.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:04:35.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:35.979 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:35.979 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:35.979 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:35.979 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:35.979 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:35.980 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:36.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:04:36.320 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:36.321 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:36.321 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:36.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:04:36.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:04:36.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:04:36.690 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:417: TEST_auto_repair_bluestore_tag: get_pg testpool SOMETHING 2024-09-18T15:04:36.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-09-18T15:04:36.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T15:04:36.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-09-18T15:04:36.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:04:37.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:417: TEST_auto_repair_bluestore_tag: local pgid=1.0 2024-09-18T15:04:37.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:418: TEST_auto_repair_bluestore_tag: get_primary testpool SOMETHING 2024-09-18T15:04:37.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-09-18T15:04:37.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T15:04:37.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T15:04:37.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:04:37.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:418: TEST_auto_repair_bluestore_tag: local primary=1 2024-09-18T15:04:37.404 INFO:tasks.workunit.client.0.smithi046.stdout:Affected PG 1.0 w/ primary 1 2024-09-18T15:04:37.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:419: TEST_auto_repair_bluestore_tag: echo 'Affected PG ' 1.0 ' w/ primary ' 1 2024-09-18T15:04:37.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:420: TEST_auto_repair_bluestore_tag: get_last_scrub_stamp 1.0 2024-09-18T15:04:37.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:04:37.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:04:37.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:04:37.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:04:37.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:420: TEST_auto_repair_bluestore_tag: local last_scrub_stamp=2024-09-18T15:04:18.274003+0000 2024-09-18T15:04:37.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:421: TEST_auto_repair_bluestore_tag: initiate_and_fetch_state 1 1.0 3.0 2024-09-18T15:04:37.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:325: initiate_and_fetch_state: local the_osd=osd.1 2024-09-18T15:04:37.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:326: initiate_and_fetch_state: local pgid=1.0 2024-09-18T15:04:37.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:327: initiate_and_fetch_state: get_last_scrub_stamp 1.0 2024-09-18T15:04:37.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:04:37.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:04:37.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:04:37.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:04:38.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:327: initiate_and_fetch_state: local last_scrub=2024-09-18T15:04:18.274003+0000 2024-09-18T15:04:38.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:329: initiate_and_fetch_state: set_config osd 1 osd_scrub_sleep 3.0 2024-09-18T15:04:38.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:04:38.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T15:04:38.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_sleep 2024-09-18T15:04:38.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=3.0 2024-09-18T15:04:38.003 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:04:38.003 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T15:04:38.003 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:04:38.003 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:04:38.003 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:04:38.003 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:04:38.003 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:04:38.003 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-osd.1.asok 2024-09-18T15:04:38.003 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.137209/ceph-osd.1.asok config set osd_scrub_sleep 3.0 2024-09-18T15:04:38.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:04:38.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:330: initiate_and_fetch_state: set_config osd 1 osd_scrub_auto_repair true 2024-09-18T15:04:38.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:04:38.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T15:04:38.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_auto_repair 2024-09-18T15:04:38.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:04:38.093 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:04:38.094 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T15:04:38.094 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:04:38.095 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:04:38.096 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:04:38.096 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:04:38.096 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:04:38.096 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-osd.1.asok 2024-09-18T15:04:38.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.137209/ceph-osd.1.asok config set osd_scrub_auto_repair true 2024-09-18T15:04:38.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:04:38.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:332: initiate_and_fetch_state: flush_pg_stats 2024-09-18T15:04:38.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:04:38.186 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:04:38.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:04:38.485 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:04:38.485 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:04:38.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:04:38.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:04:38.486 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:04:38.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182404 2024-09-18T15:04:38.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182404 2024-09-18T15:04:38.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182404' 2024-09-18T15:04:38.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:04:38.635 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:04:38.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672971 2024-09-18T15:04:38.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672971 2024-09-18T15:04:38.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182404 1-42949672971' 2024-09-18T15:04:38.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:04:38.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:04:38.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509449 2024-09-18T15:04:38.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509449 2024-09-18T15:04:38.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182404 1-42949672971 2-64424509449' 2024-09-18T15:04:38.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:04:38.917 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-107374182404 2024-09-18T15:04:38.918 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:04:38.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:04:38.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-107374182404 2024-09-18T15:04:38.921 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:04:38.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182404 2024-09-18T15:04:38.923 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 107374182404 2024-09-18T15:04:38.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 107374182404' 2024-09-18T15:04:38.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:04:39.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182404 -lt 107374182404 2024-09-18T15:04:39.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:04:39.225 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672971 2024-09-18T15:04:39.225 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:04:39.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:04:39.228 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672971 2024-09-18T15:04:39.229 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:04:39.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672971 2024-09-18T15:04:39.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-09-18T15:04:39.230 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672971 2024-09-18T15:04:39.231 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:04:39.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672970 -lt 42949672971 2024-09-18T15:04:39.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:04:40.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:04:40.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:04:40.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672971 -lt 42949672971 2024-09-18T15:04:40.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:04:40.925 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509449 2024-09-18T15:04:40.925 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:04:40.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:04:40.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509449 2024-09-18T15:04:40.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:04:40.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509449 2024-09-18T15:04:40.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-18T15:04:40.930 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509449 2024-09-18T15:04:40.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:04:41.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509449 -lt 64424509449 2024-09-18T15:04:41.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: date --rfc-3339=ns 2024-09-18T15:04:41.274 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18 15:04:41.270191038+00:00 2024-09-18T15:04:41.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: get_asok_path osd.1 2024-09-18T15:04:41.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:04:41.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:04:41.276 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:04:41.276 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:04:41.276 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:04:41.277 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-osd.1.asok 2024-09-18T15:04:41.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.137209/ceph-osd.1.asok deep_scrub 1.0 2024-09-18T15:04:41.374 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:337: initiate_and_fetch_state: get_asok_path osd.1 2024-09-18T15:04:41.374 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:04:41.374 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:04:41.375 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:04:41.375 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:04:41.375 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:04:41.375 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-osd.1.asok 2024-09-18T15:04:41.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:337: initiate_and_fetch_state: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.137209/ceph-osd.1.asok scrub 1.0 2024-09-18T15:04:41.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i=0 )) 2024-09-18T15:04:41.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i < 80 )) 2024-09-18T15:04:41.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-18T15:04:41.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: jq .state 2024-09-18T15:04:41.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: st='"active+clean"' 2024-09-18T15:04:41.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 0 ') state now: ' '"active+clean"' 2024-09-18T15:04:41.613 INFO:tasks.workunit.client.0.smithi046.stdout:{"deep":true,"stamp":"2024-09-11T15:03:01.360638+0000"}{"deep":false,"stamp":"2024-09-11T15:03:01.461597+0000"}0 ) state now: "active+clean" 2024-09-18T15:04:41.613 INFO:tasks.workunit.client.0.smithi046.stdout:"active+clean" 2024-09-18T15:04:41.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:345: initiate_and_fetch_state: case "$st" in 2024-09-18T15:04:41.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: initiate_and_fetch_state: echo '"active+clean"' 2024-09-18T15:04:41.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:353: initiate_and_fetch_state: '[' 0 == 4 ']' 2024-09-18T15:04:41.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:356: initiate_and_fetch_state: sleep 0.3 2024-09-18T15:04:41.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i++ )) 2024-09-18T15:04:41.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i < 80 )) 2024-09-18T15:04:41.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-18T15:04:41.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: jq .state 2024-09-18T15:04:42.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: st='"active+clean"' 2024-09-18T15:04:42.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 1 ') state now: ' '"active+clean"' 2024-09-18T15:04:42.062 INFO:tasks.workunit.client.0.smithi046.stdout:1 ) state now: "active+clean" 2024-09-18T15:04:42.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:345: initiate_and_fetch_state: case "$st" in 2024-09-18T15:04:42.063 INFO:tasks.workunit.client.0.smithi046.stdout:"active+clean" 2024-09-18T15:04:42.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: initiate_and_fetch_state: echo '"active+clean"' 2024-09-18T15:04:42.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:353: initiate_and_fetch_state: '[' 1 == 4 ']' 2024-09-18T15:04:42.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:356: initiate_and_fetch_state: sleep 0.3 2024-09-18T15:04:42.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i++ )) 2024-09-18T15:04:42.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: initiate_and_fetch_state: (( i < 80 )) 2024-09-18T15:04:42.365 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-18T15:04:42.365 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: jq .state 2024-09-18T15:04:42.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: initiate_and_fetch_state: st='"active+clean+scrubbing+deep"' 2024-09-18T15:04:42.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 2 ') state now: ' '"active+clean+scrubbing+deep"' 2024-09-18T15:04:42.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:345: initiate_and_fetch_state: case "$st" in 2024-09-18T15:04:42.511 INFO:tasks.workunit.client.0.smithi046.stdout:2 ) state now: "active+clean+scrubbing+deep" 2024-09-18T15:04:42.511 INFO:tasks.workunit.client.0.smithi046.stdout:found scrub 2024-09-18T15:04:42.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:347: initiate_and_fetch_state: echo 'found scrub' 2024-09-18T15:04:42.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:347: initiate_and_fetch_state: return 0 2024-09-18T15:04:42.512 INFO:tasks.workunit.client.0.smithi046.stdout:initiate_and_fetch_state ret: 0 2024-09-18T15:04:42.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:422: TEST_auto_repair_bluestore_tag: r=0 2024-09-18T15:04:42.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: TEST_auto_repair_bluestore_tag: echo 'initiate_and_fetch_state ret: ' 0 2024-09-18T15:04:42.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: TEST_auto_repair_bluestore_tag: set_config osd td/osd-scrub-repair osd_scrub_sleep 0 2024-09-18T15:04:42.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:04:42.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=td/osd-scrub-repair 2024-09-18T15:04:42.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_sleep 2024-09-18T15:04:42.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=0 2024-09-18T15:04:42.513 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:04:42.513 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.td/osd-scrub-repair 2024-09-18T15:04:42.514 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.td/osd-scrub-repair 2024-09-18T15:04:42.514 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.td/osd-scrub-repair ']' 2024-09-18T15:04:42.514 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:04:42.515 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:04:42.515 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:04:42.515 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-osd.td/osd-scrub-repair.asok 2024-09-18T15:04:42.515 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.137209/ceph-osd.td/osd-scrub-repair.asok config set osd_scrub_sleep 0 2024-09-18T15:04:42.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test == true 2024-09-18T15:04:42.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 1168: test: ==: unary operator expected 2024-09-18T15:04:42.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: TEST_auto_repair_bluestore_tag: '[' 0 -ne 0 ']' 2024-09-18T15:04:42.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:429: TEST_auto_repair_bluestore_tag: wait_end_of_scrub 1 1.0 2024-09-18T15:04:42.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: local the_osd=osd.1 2024-09-18T15:04:42.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: local pgid=1.0 2024-09-18T15:04:42.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: (( i=0 )) 2024-09-18T15:04:42.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: (( i < 40 )) 2024-09-18T15:04:42.593 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-18T15:04:42.594 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: jq .state 2024-09-18T15:04:42.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-09-18T15:04:42.786 INFO:tasks.workunit.client.0.smithi046.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-09-18T15:04:42.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:369: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-09-18T15:04:42.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-09-18T15:04:42.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:371: wait_end_of_scrub: '[' 0 == 4 ']' 2024-09-18T15:04:42.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:374: wait_end_of_scrub: sleep 0.3 2024-09-18T15:04:43.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: (( i++ )) 2024-09-18T15:04:43.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: (( i < 40 )) 2024-09-18T15:04:43.089 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-18T15:04:43.089 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: jq .state 2024-09-18T15:04:43.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-09-18T15:04:43.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:369: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-09-18T15:04:43.250 INFO:tasks.workunit.client.0.smithi046.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-09-18T15:04:43.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-09-18T15:04:43.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:371: wait_end_of_scrub: '[' 1 == 4 ']' 2024-09-18T15:04:43.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:374: wait_end_of_scrub: sleep 0.3 2024-09-18T15:04:43.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: (( i++ )) 2024-09-18T15:04:43.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: (( i < 40 )) 2024-09-18T15:04:43.553 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-18T15:04:43.553 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: jq .state 2024-09-18T15:04:43.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-09-18T15:04:43.698 INFO:tasks.workunit.client.0.smithi046.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-09-18T15:04:43.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:369: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-09-18T15:04:43.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-09-18T15:04:43.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:371: wait_end_of_scrub: '[' 2 == 4 ']' 2024-09-18T15:04:43.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:374: wait_end_of_scrub: sleep 0.3 2024-09-18T15:04:43.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: (( i++ )) 2024-09-18T15:04:44.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: (( i < 40 )) 2024-09-18T15:04:44.001 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-18T15:04:44.001 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: jq .state 2024-09-18T15:04:44.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-09-18T15:04:44.153 INFO:tasks.workunit.client.0.smithi046.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-09-18T15:04:44.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:369: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-09-18T15:04:44.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-09-18T15:04:44.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:371: wait_end_of_scrub: '[' 3 == 4 ']' 2024-09-18T15:04:44.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:374: wait_end_of_scrub: sleep 0.3 2024-09-18T15:04:44.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: (( i++ )) 2024-09-18T15:04:44.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: (( i < 40 )) 2024-09-18T15:04:44.455 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-18T15:04:44.455 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: jq .state 2024-09-18T15:04:44.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-09-18T15:04:44.604 INFO:tasks.workunit.client.0.smithi046.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-09-18T15:04:44.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:369: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-09-18T15:04:44.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-09-18T15:04:44.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:371: wait_end_of_scrub: '[' 4 == 4 ']' 2024-09-18T15:04:44.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:372: wait_end_of_scrub: flush_pg_stats 2024-09-18T15:04:44.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:04:44.605 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:04:44.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:04:44.912 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:04:44.912 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:04:44.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:04:44.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:04:44.912 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:04:45.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182406 2024-09-18T15:04:45.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182406 2024-09-18T15:04:45.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182406' 2024-09-18T15:04:45.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:04:45.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:04:45.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672973 2024-09-18T15:04:45.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672973 2024-09-18T15:04:45.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182406 1-42949672973' 2024-09-18T15:04:45.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:04:45.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:04:45.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509451 2024-09-18T15:04:45.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509451 2024-09-18T15:04:45.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182406 1-42949672973 2-64424509451' 2024-09-18T15:04:45.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:04:45.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-107374182406 2024-09-18T15:04:45.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:04:45.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:04:45.444 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-107374182406 2024-09-18T15:04:45.444 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:04:45.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182406 2024-09-18T15:04:45.446 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 107374182406 2024-09-18T15:04:45.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 107374182406' 2024-09-18T15:04:45.447 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:04:45.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182405 -lt 107374182406 2024-09-18T15:04:45.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:04:46.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:04:46.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:04:47.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182407 -lt 107374182406 2024-09-18T15:04:47.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:04:47.080 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672973 2024-09-18T15:04:47.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:04:47.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:04:47.083 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672973 2024-09-18T15:04:47.084 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:04:47.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672973 2024-09-18T15:04:47.086 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672973 2024-09-18T15:04:47.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-09-18T15:04:47.087 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:04:47.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672974 -lt 42949672973 2024-09-18T15:04:47.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:04:47.503 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509451 2024-09-18T15:04:47.503 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:04:47.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:04:47.507 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509451 2024-09-18T15:04:47.507 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:04:47.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509451 2024-09-18T15:04:47.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-18T15:04:47.509 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509451 2024-09-18T15:04:47.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:04:47.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509451 -lt 64424509451 2024-09-18T15:04:47.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:374: wait_end_of_scrub: sleep 0.3 2024-09-18T15:04:48.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: (( i++ )) 2024-09-18T15:04:48.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: (( i < 40 )) 2024-09-18T15:04:48.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-18T15:04:48.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: jq .state 2024-09-18T15:04:48.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: st='"active+clean"' 2024-09-18T15:04:48.273 INFO:tasks.workunit.client.0.smithi046.stdout:wait-scrub-end state now: "active+clean" 2024-09-18T15:04:48.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:369: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean"' 2024-09-18T15:04:48.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-09-18T15:04:48.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: break 2024-09-18T15:04:48.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:377: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-09-18T15:04:48.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:382: wait_end_of_scrub: return 0 2024-09-18T15:04:48.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: ceph pg dump pgs 2024-09-18T15:04:48.562 INFO:tasks.workunit.client.0.smithi046.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T15:04:48.562 INFO:tasks.workunit.client.0.smithi046.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-09-18T15:04:45.407397+0000 21'1 26:62 [1,0] 1 [1,0] 1 21'1 2024-09-18T15:04:45.113450+0000 21'1 2024-09-18T15:04:45.113450+0000 0 4 periodic scrub scheduled @ 2024-09-19T15:04:45.113450+0000 1 0 2024-09-18T15:04:48.563 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-09-18T15:04:48.563 INFO:tasks.workunit.client.0.smithi046.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T15:04:48.566 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T15:04:48.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:434: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-09-18T15:04:48.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-09-18T15:04:48.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-09-18T15:04:48.576 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-09-18T15:04:48.576 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-09-18T15:04:48.576 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T15:04:48.577 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T15:04:48.577 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:04:48.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T15:04:48.914 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T15:04:48.914 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T15:04:49.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:434: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-18T15:04:49.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:04:49.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:04:49.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:04:49.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:04:49.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-18T15:04:49.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:04:49.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:04:49.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:04:49.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:04:49.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:04:49.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:04:49.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:04:49.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:04:49.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:04:49.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:04:49.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:04:49.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-18T15:04:49.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:04:49.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:04:49.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:04:49.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:04:49.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:04:49.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-09-18T15:04:51.003 INFO:tasks.workunit.client.0.smithi046.stdout:_ 2024-09-18T15:04:51.003 INFO:tasks.workunit.client.0.smithi046.stdout:snapset 2024-09-18T15:04:51.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-18T15:04:51.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:04:51.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:04:51.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:04:51.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:04:51.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:04:51.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:04:51.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:04:51.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:04:51.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:04:51.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:04:51.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:04:51.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:04:51.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:04:51.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:04:51.541 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:04:51.541 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:04:51.541 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:04:51.541 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:04:51.541 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:04:51.541 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:04:51.541 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:04:51.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:04:51.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:04:51.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:04:51.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:04:51.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:04:51.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:04:51.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:04:51.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:04:51.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:04:51.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-18T15:04:51.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:04:51.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:04:51.544 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:04:51.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-18T15:04:51.545 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:04:51.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:04:51.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:04:51.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:04:51.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:04:51.581 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:51.574+0000 7fe083d26540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:04:51.585 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:51.578+0000 7fe083d26540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:04:51.586 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:51.578+0000 7fe083d26540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:04:51.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:04:51.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:04:51.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:04:51.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:04:51.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:04:51.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:04:51.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:04:51.858 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:04:51.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:04:51.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:04:52.127 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:52.118+0000 7fe083d26540 -1 Falling back to public interface 2024-09-18T15:04:52.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:04:53.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:04:53.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:04:53.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:04:53.217 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:04:53.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:04:53.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:04:53.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:04:54.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:04:54.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:04:54.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:04:54.535 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:04:54.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:04:54.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:04:54.773 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:04:54.766+0000 7fe083d26540 -1 osd.0 26 log_to_monitors true 2024-09-18T15:04:54.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:04:55.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:04:55.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:04:55.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:04:55.849 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:04:55.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:04:55.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:04:56.156 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/200401,v1:127.0.0.1:6803/200401] [v2:127.0.0.1:6804/200401,v1:127.0.0.1:6805/200401] exists,up f917995b-d31d-42dd-ab17-1bf7b28590db 2024-09-18T15:04:56.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:04:56.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:04:56.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:04:56.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:04:56.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:04:56.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:04:56.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:04:56.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:04:56.159 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:04:56.159 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:04:56.159 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:04:56.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:04:56.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:04:56.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:04:56.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:04:56.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:04:56.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:04:56.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:04:56.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:04:56.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:04:56.690 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:04:56.690 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:04:56.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:04:56.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:04:56.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:04:56.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018882 2024-09-18T15:04:56.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018882 2024-09-18T15:04:56.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-128849018882' 2024-09-18T15:04:56.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:04:56.873 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:04:57.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672977 2024-09-18T15:04:57.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672977 2024-09-18T15:04:57.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-128849018882 1-42949672977' 2024-09-18T15:04:57.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:04:57.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:04:57.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509454 2024-09-18T15:04:57.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509454 2024-09-18T15:04:57.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-128849018882 1-42949672977 2-64424509454' 2024-09-18T15:04:57.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:04:57.151 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-128849018882 2024-09-18T15:04:57.152 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:04:57.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:04:57.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-128849018882 2024-09-18T15:04:57.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:04:57.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018882 2024-09-18T15:04:57.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-09-18T15:04:57.157 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 128849018882 2024-09-18T15:04:57.157 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:04:57.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 128849018882 2024-09-18T15:04:57.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:04:58.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:04:58.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:04:58.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018882 -lt 128849018882 2024-09-18T15:04:58.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:04:58.796 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672977 2024-09-18T15:04:58.796 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:04:58.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:04:58.799 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672977 2024-09-18T15:04:58.799 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:04:58.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672977 2024-09-18T15:04:58.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-18T15:04:58.802 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 42949672977 2024-09-18T15:04:58.802 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:04:59.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-18T15:04:59.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:04:59.149 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509454 2024-09-18T15:04:59.150 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:04:59.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:04:59.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509454 2024-09-18T15:04:59.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:04:59.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509454 2024-09-18T15:04:59.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-09-18T15:04:59.156 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509454 2024-09-18T15:04:59.156 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:04:59.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509455 -lt 64424509454 2024-09-18T15:04:59.492 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:04:59.492 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:59.492 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:59.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:04:59.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:59.861 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:59.861 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:59.861 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:59.861 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:59.861 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:59.861 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:00.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:00.203 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:00.203 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:00.203 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:00.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:05:00.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:05:00.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:05:00.568 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:435: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-09-18T15:05:00.568 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-09-18T15:05:00.568 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-09-18T15:05:00.569 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-09-18T15:05:00.569 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-09-18T15:05:00.569 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T15:05:00.569 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T15:05:00.569 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:05:00.946 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T15:05:00.946 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-18T15:05:00.947 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T15:05:01.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:435: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-18T15:05:01.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:05:01.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:05:01.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:05:01.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:05:01.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-18T15:05:01.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:05:01.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:05:01.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:05:01.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:05:01.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:05:01.265 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:05:01.265 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:05:01.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:05:01.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:05:01.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:05:01.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:05:01.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-18T15:05:01.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:05:01.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:05:01.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:05:01.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:05:01.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:05:01.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-18T15:05:03.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-18T15:05:03.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:05:03.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:05:03.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:05:03.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:05:03.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:05:03.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:05:03.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:05:03.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:05:03.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:05:03.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:05:03.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:05:03.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:05:03.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:05:03.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:05:03.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:05:03.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:05:03.622 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:05:03.623 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:05:03.623 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:05:03.623 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:05:03.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:05:03.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:05:03.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:05:03.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:05:03.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:05:03.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:05:03.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:05:03.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:05:03.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:05:03.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:05:03.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-18T15:05:03.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:05:03.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:05:03.625 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:05:03.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-18T15:05:03.626 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:05:03.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:05:03.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:05:03.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:05:03.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:05:03.659 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:03.650+0000 7f6052e36540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:05:03.664 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:03.654+0000 7f6052e36540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:05:03.665 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:03.658+0000 7f6052e36540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:05:03.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:05:03.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:05:03.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:05:03.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:05:03.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:05:03.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:05:03.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:05:03.940 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:05:03.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:05:03.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:05:04.207 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:04.198+0000 7f6052e36540 -1 Falling back to public interface 2024-09-18T15:05:04.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:05:05.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:05:05.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:05:05.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:05:05.279 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:05:05.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:05:05.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:05:05.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:05:06.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:05:06.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:05:06.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:05:06.601 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:05:06.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:05:06.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:05:06.917 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:06.910+0000 7f6052e36540 -1 osd.0 31 log_to_monitors true 2024-09-18T15:05:07.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:05:08.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:05:08.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:05:08.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:05:08.008 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:05:08.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:05:08.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:05:08.321 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/201803,v1:127.0.0.1:6803/201803] [v2:127.0.0.1:6804/201803,v1:127.0.0.1:6805/201803] exists,up f917995b-d31d-42dd-ab17-1bf7b28590db 2024-09-18T15:05:08.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:05:08.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:05:08.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:05:08.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:05:08.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:05:08.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:05:08.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:05:08.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:05:08.323 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:05:08.324 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:05:08.324 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:05:08.324 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:05:08.324 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:05:08.324 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:05:08.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:05:08.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:05:08.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:05:08.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:05:08.538 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:05:08.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:05:08.849 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:05:08.849 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:05:08.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:05:08.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:05:08.849 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:05:08.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855362 2024-09-18T15:05:08.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855362 2024-09-18T15:05:08.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855362' 2024-09-18T15:05:08.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:05:08.991 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:05:09.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672980 2024-09-18T15:05:09.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672980 2024-09-18T15:05:09.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855362 1-42949672980' 2024-09-18T15:05:09.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:05:09.136 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:05:09.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509458 2024-09-18T15:05:09.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509458 2024-09-18T15:05:09.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855362 1-42949672980 2-64424509458' 2024-09-18T15:05:09.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:05:09.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-150323855362 2024-09-18T15:05:09.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:05:09.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:05:09.283 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-150323855362 2024-09-18T15:05:09.283 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:05:09.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855362 2024-09-18T15:05:09.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-09-18T15:05:09.285 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 150323855362 2024-09-18T15:05:09.286 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:05:09.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 150323855362 2024-09-18T15:05:09.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:05:10.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:05:10.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:05:10.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855362 -lt 150323855362 2024-09-18T15:05:10.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:05:10.942 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672980 2024-09-18T15:05:10.942 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:05:10.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:05:10.945 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672980 2024-09-18T15:05:10.946 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:05:10.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672980 2024-09-18T15:05:10.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-09-18T15:05:10.947 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 42949672980 2024-09-18T15:05:10.948 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:05:11.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672980 -lt 42949672980 2024-09-18T15:05:11.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:05:11.285 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509458 2024-09-18T15:05:11.285 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:05:11.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:05:11.287 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509458 2024-09-18T15:05:11.288 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:05:11.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509458 2024-09-18T15:05:11.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-09-18T15:05:11.290 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509458 2024-09-18T15:05:11.290 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:05:11.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509458 -lt 64424509458 2024-09-18T15:05:11.637 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:05:11.637 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:11.638 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:12.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:05:12.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:12.056 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:12.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:12.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:12.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:12.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:12.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:12.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:12.402 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:12.402 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:12.402 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:12.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:05:12.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:05:12.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:05:12.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:436: TEST_auto_repair_bluestore_tag: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-18T15:05:12.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:437: TEST_auto_repair_bluestore_tag: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-18T15:05:12.771 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T15:04:45.110+0000 7fe4dec99700 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber : scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-18T15:05:12.771 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T15:04:45.110+0000 7fe4dec99700 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber : _scrub_finish info stats: valid 2024-09-18T15:05:12.771 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T15:04:45.110+0000 7fe4dec99700 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber : scrub_finish All may be fixed 2024-09-18T15:05:12.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T15:05:12.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:05:12.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:05:12.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:05:12.772 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:05:12.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:05:12.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:05:12.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:05:12.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:05:12.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:05:12.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:05:12.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:05:12.890 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:05:12.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:05:12.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:05:12.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:05:12.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:05:12.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:05:12.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:05:12.897 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:05:12.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:05:12.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:05:12.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:05:12.912 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:05:12.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:05:12.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:05:12.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:05:12.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:05:12.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:05:12.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T15:05:12.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T15:05:12.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T15:05:12.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T15:05:12.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:05:12.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:05:12.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:05:12.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:05:12.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:05:12.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:05:12.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:05:12.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:05:12.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:05:12.921 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:05:12.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:05:12.922 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:05:12.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:05:12.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:05:12.925 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:05:12.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:05:12.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:05:12.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:05:12.927 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:05:12.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:05:12.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:05:12.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:05:12.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:05:12.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:05:12.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:05:12.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:05:12.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:05:12.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:05:12.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T15:05:12.936 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:05:12.936 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:05:12.937 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:05:12.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T15:05:12.939 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:05:12.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:05:12.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:05:12.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T15:05:12.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_auto_repair_erasure_coded_appends td/osd-scrub-repair 2024-09-18T15:05:12.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:312: TEST_auto_repair_erasure_coded_appends: auto_repair_erasure_coded td/osd-scrub-repair false 2024-09-18T15:05:12.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:267: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-18T15:05:12.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local allow_overwrites=false 2024-09-18T15:05:12.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local poolname=ecpool 2024-09-18T15:05:12.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-09-18T15:05:12.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T15:05:12.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:05:12.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:05:12.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:05:12.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T15:05:12.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-18T15:05:13.324 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:05:13.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:05:13.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:05:13.325 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:05:13.325 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:05:13.325 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:05:13.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:05:13.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T15:05:13.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:05:13.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:05:13.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:05:13.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:05:13.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:05:13.598 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:05:13.598 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:05:13.598 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:05:13.598 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:05:13.599 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:05:13.599 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:05:13.599 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:05:13.599 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:05:13.600 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:05:13.600 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T15:05:13.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:05:13.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:05:13.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:05:13.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:05:13.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:05:13.693 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:05:13.693 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:05:13.693 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:05:13.694 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:05:13.694 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:05:13.694 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:05:13.694 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:05:13.695 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:05:13.695 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T15:05:13.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-09-18T15:05:13.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T15:05:13.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:05:13.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:05:13.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:05:13.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T15:05:13.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:05:14.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:05:14.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:05:14.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:05:14.642 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:05:14.642 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:05:14.642 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:05:14.642 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:05:14.644 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:05:14.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:05:14.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:278: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:05:14.683 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: seq 0 2 2024-09-18T15:05:14.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-18T15:05:14.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: '[' false = true ']' 2024-09-18T15:05:14.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: run_osd_filestore td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:05:14.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T15:05:14.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T15:05:14.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=0 2024-09-18T15:05:14.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T15:05:14.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:05:14.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:05:14.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:05:14.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:05:14.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:05:14.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:05:14.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:05:14.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T15:05:14.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T15:05:14.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:05:14.687 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T15:05:14.687 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:05:14.687 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:05:14.689 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:05:14.689 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:05:14.689 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:05:14.689 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:05:14.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:05:14.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T15:05:14.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T15:05:14.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T15:05:14.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:05:14.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:05:14.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:05:14.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:05:14.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:05:14.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T15:05:14.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:05:14.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:05:14.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T15:05:14.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=83205ef5-77b9-4cbb-80e4-1196c3426640 2024-09-18T15:05:14.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd2 83205ef5-77b9-4cbb-80e4-1196c3426640' 2024-09-18T15:05:14.692 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 83205ef5-77b9-4cbb-80e4-1196c3426640 2024-09-18T15:05:14.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T15:05:14.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQAq7OpmdxgbKhAABMyZL6Y58Vkmff035WwOMg== 2024-09-18T15:05:14.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQAq7OpmdxgbKhAABMyZL6Y58Vkmff035WwOMg=="}' 2024-09-18T15:05:14.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 83205ef5-77b9-4cbb-80e4-1196c3426640 -i td/osd-scrub-repair/0/new.json 2024-09-18T15:05:14.991 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:05:15.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/0/new.json 2024-09-18T15:05:15.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAq7OpmdxgbKhAABMyZL6Y58Vkmff035WwOMg== --osd-uuid 83205ef5-77b9-4cbb-80e4-1196c3426640 --osd-objectstore=filestore 2024-09-18T15:05:15.038 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:15.030+0000 7f1a13c2f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:05:15.042 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:15.034+0000 7f1a13c2f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:05:15.044 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:15.034+0000 7f1a13c2f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:05:15.543 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:15.534+0000 7f1a13c2f540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:05:16.122 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:16.114+0000 7f1a13c2f540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:05:16.122 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:16.114+0000 7f1a13c2f540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:05:16.122 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:16.114+0000 7f1a13c2f540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:05:16.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T15:05:16.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T15:05:16.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd0 key to auth repository 2024-09-18T15:05:16.649 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T15:05:16.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:05:16.827 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: key for osd.0 not found in provided keyring 2024-09-18T15:05:16.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.0 2024-09-18T15:05:16.829 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T15:05:16.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:05:16.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T15:05:16.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T15:05:16.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T15:05:16.857 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:16.850+0000 7f2f68f0a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:05:16.859 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:16.850+0000 7f2f68f0a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:05:16.860 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:16.854+0000 7f2f68f0a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:05:16.862 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:16.854+0000 7f2f68f0a540 -1 Falling back to public interface 2024-09-18T15:05:17.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 0 2024-09-18T15:05:17.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:05:17.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:05:17.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:05:17.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:05:17.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:05:17.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:05:17.166 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:05:17.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:05:17.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:05:17.429 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:17.418+0000 7f2f68f0a540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:05:17.429 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:17.422+0000 7f2f68f0a540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:05:17.429 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:17.422+0000 7f2f68f0a540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:05:17.435 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:17.426+0000 7f2f68f0a540 -1 osd.0 0 log_to_monitors true 2024-09-18T15:05:17.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:05:18.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:05:18.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:05:18.479 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:05:18.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:05:18.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:05:18.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:05:18.659 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/203081,v1:127.0.0.1:6801/203081] [v2:127.0.0.1:6802/203081,v1:127.0.0.1:6803/203081] exists,up 83205ef5-77b9-4cbb-80e4-1196c3426640 2024-09-18T15:05:18.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:05:18.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:05:18.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:05:18.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-18T15:05:18.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: '[' false = true ']' 2024-09-18T15:05:18.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: run_osd_filestore td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:05:18.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T15:05:18.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T15:05:18.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=1 2024-09-18T15:05:18.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T15:05:18.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:05:18.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:05:18.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:05:18.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:05:18.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:05:18.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:05:18.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:05:18.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T15:05:18.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T15:05:18.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:05:18.664 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T15:05:18.664 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:05:18.664 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:05:18.664 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:05:18.664 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:05:18.664 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:05:18.664 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:05:18.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:05:18.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T15:05:18.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T15:05:18.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T15:05:18.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:05:18.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:05:18.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:05:18.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:05:18.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:05:18.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T15:05:18.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:05:18.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:05:18.667 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T15:05:18.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=1517c6db-ecb0-4772-9b8a-558021bd9092 2024-09-18T15:05:18.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd2 1517c6db-ecb0-4772-9b8a-558021bd9092' 2024-09-18T15:05:18.667 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 1517c6db-ecb0-4772-9b8a-558021bd9092 2024-09-18T15:05:18.667 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T15:05:18.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQAu7OpmJaSmKBAAsI297BgNcAfprdTZ3p8vFQ== 2024-09-18T15:05:18.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQAu7OpmJaSmKBAAsI297BgNcAfprdTZ3p8vFQ=="}' 2024-09-18T15:05:18.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 1517c6db-ecb0-4772-9b8a-558021bd9092 -i td/osd-scrub-repair/1/new.json 2024-09-18T15:05:18.952 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:05:18.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/1/new.json 2024-09-18T15:05:18.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAu7OpmJaSmKBAAsI297BgNcAfprdTZ3p8vFQ== --osd-uuid 1517c6db-ecb0-4772-9b8a-558021bd9092 --osd-objectstore=filestore 2024-09-18T15:05:18.995 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:18.986+0000 7f5fb8d8a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:05:19.001 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:18.994+0000 7f5fb8d8a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:05:19.003 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:18.994+0000 7f5fb8d8a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:05:19.538 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:19.530+0000 7f5fb8d8a540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:05:20.167 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:20.158+0000 7f5fb8d8a540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:05:20.174 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:20.166+0000 7f5fb8d8a540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:05:20.175 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:20.166+0000 7f5fb8d8a540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:05:21.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-18T15:05:21.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T15:05:21.223 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T15:05:21.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd1 key to auth repository 2024-09-18T15:05:21.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:05:21.630 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: key for osd.1 not found in provided keyring 2024-09-18T15:05:21.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.1 2024-09-18T15:05:21.633 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T15:05:21.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:05:21.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T15:05:21.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T15:05:21.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T15:05:21.661 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:21.654+0000 7f98865e0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:05:21.663 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:21.654+0000 7f98865e0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:05:21.664 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:21.658+0000 7f98865e0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:05:21.666 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:21.658+0000 7f98865e0540 -1 Falling back to public interface 2024-09-18T15:05:22.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 1 2024-09-18T15:05:22.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:05:22.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:05:22.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:05:22.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:05:22.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:05:22.040 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:05:22.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:05:22.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:05:22.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:05:22.219 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:22.210+0000 7f98865e0540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:05:22.226 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:22.218+0000 7f98865e0540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:05:22.227 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:22.218+0000 7f98865e0540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:05:22.236 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:22.226+0000 7f98865e0540 -1 osd.1 0 log_to_monitors true 2024-09-18T15:05:22.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:05:23.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:05:23.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:05:23.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:05:23.349 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:05:23.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:05:23.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:05:23.711 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/203463,v1:127.0.0.1:6811/203463] [v2:127.0.0.1:6812/203463,v1:127.0.0.1:6813/203463] exists,up 1517c6db-ecb0-4772-9b8a-558021bd9092 2024-09-18T15:05:23.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:05:23.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:05:23.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:05:23.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-18T15:05:23.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: '[' false = true ']' 2024-09-18T15:05:23.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: run_osd_filestore td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:05:23.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T15:05:23.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T15:05:23.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=2 2024-09-18T15:05:23.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T15:05:23.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:05:23.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:05:23.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:05:23.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:05:23.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:05:23.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T15:05:23.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T15:05:23.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T15:05:23.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T15:05:23.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:05:23.715 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T15:05:23.715 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:05:23.715 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:05:23.717 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:05:23.717 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:05:23.717 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:05:23.717 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:05:23.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:05:23.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T15:05:23.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T15:05:23.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T15:05:23.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:05:23.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:05:23.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:05:23.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:05:23.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:05:23.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T15:05:23.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:05:23.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/2 2024-09-18T15:05:23.718 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T15:05:23.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=9fb2f716-97c2-4c67-a912-28c26c78815a 2024-09-18T15:05:23.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd2 9fb2f716-97c2-4c67-a912-28c26c78815a' 2024-09-18T15:05:23.720 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 9fb2f716-97c2-4c67-a912-28c26c78815a 2024-09-18T15:05:23.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T15:05:23.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQAz7OpmUtn1KxAAmCgoH4oirMEZGdDMEKAmCQ== 2024-09-18T15:05:23.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQAz7OpmUtn1KxAAmCgoH4oirMEZGdDMEKAmCQ=="}' 2024-09-18T15:05:23.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 9fb2f716-97c2-4c67-a912-28c26c78815a -i td/osd-scrub-repair/2/new.json 2024-09-18T15:05:24.171 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:05:24.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/2/new.json 2024-09-18T15:05:24.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAz7OpmUtn1KxAAmCgoH4oirMEZGdDMEKAmCQ== --osd-uuid 9fb2f716-97c2-4c67-a912-28c26c78815a --osd-objectstore=filestore 2024-09-18T15:05:24.228 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:24.218+0000 7fa5317c5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:05:24.237 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:24.230+0000 7fa5317c5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:05:24.239 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:24.230+0000 7fa5317c5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:05:25.040 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:25.034+0000 7fa5317c5540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:05:25.737 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:25.730+0000 7fa5317c5540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:05:25.738 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:25.730+0000 7fa5317c5540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:05:25.738 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:25.730+0000 7fa5317c5540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:05:26.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-18T15:05:26.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T15:05:26.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd2 key to auth repository 2024-09-18T15:05:26.543 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T15:05:26.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:05:26.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.2 2024-09-18T15:05:26.928 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T15:05:26.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T15:05:26.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:05:26.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T15:05:26.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T15:05:26.962 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:26.954+0000 7f0c80931540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:05:26.964 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:26.958+0000 7f0c80931540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:05:26.966 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:26.958+0000 7f0c80931540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:05:26.968 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:26.958+0000 7f0c80931540 -1 Falling back to public interface 2024-09-18T15:05:27.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 2 2024-09-18T15:05:27.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:05:27.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:05:27.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:05:27.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:05:27.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:05:27.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:05:27.338 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:05:27.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:05:27.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:05:27.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:05:27.747 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:27.738+0000 7f0c80931540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:05:27.764 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:27.758+0000 7f0c80931540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:05:27.764 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:27.758+0000 7f0c80931540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:05:27.801 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:27.794+0000 7f0c80931540 -1 osd.2 0 log_to_monitors true 2024-09-18T15:05:28.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:05:28.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:05:28.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:05:28.704 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:05:28.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:05:28.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:05:29.107 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/203747,v1:127.0.0.1:6819/203747] [v2:127.0.0.1:6820/203747,v1:127.0.0.1:6821/203747] exists,up 9fb2f716-97c2-4c67-a912-28c26c78815a 2024-09-18T15:05:29.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:05:29.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:05:29.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:05:29.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: auto_repair_erasure_coded: create_rbd_pool 2024-09-18T15:05:29.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T15:05:29.406 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' does not exist 2024-09-18T15:05:29.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T15:05:29.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T15:05:29.941 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' created 2024-09-18T15:05:29.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:05:30.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T15:05:33.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: wait_for_clean 2024-09-18T15:05:33.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:05:33.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:05:33.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:05:33.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:05:33.430 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:05:33.430 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:05:33.431 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:05:33.431 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:05:33.431 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:05:33.431 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:05:33.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:05:33.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:05:33.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:05:33.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:05:33.660 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:05:33.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:05:33.975 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:05:33.975 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:05:33.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:05:33.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:05:33.975 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:05:34.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836484 2024-09-18T15:05:34.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836484 2024-09-18T15:05:34.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836484' 2024-09-18T15:05:34.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:05:34.189 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:05:34.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672964 2024-09-18T15:05:34.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672964 2024-09-18T15:05:34.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836484 1-42949672964' 2024-09-18T15:05:34.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:05:34.336 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:05:34.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=68719476739 2024-09-18T15:05:34.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 68719476739 2024-09-18T15:05:34.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836484 1-42949672964 2-68719476739' 2024-09-18T15:05:34.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:05:34.476 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836484 2024-09-18T15:05:34.477 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:05:34.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:05:34.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836484 2024-09-18T15:05:34.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:05:34.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836484 2024-09-18T15:05:34.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2024-09-18T15:05:34.482 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836484 2024-09-18T15:05:34.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:05:34.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836483 -lt 21474836484 2024-09-18T15:05:34.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:05:35.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:05:35.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:05:36.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836485 -lt 21474836484 2024-09-18T15:05:36.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:05:36.168 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672964 2024-09-18T15:05:36.168 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:05:36.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:05:36.172 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672964 2024-09-18T15:05:36.172 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:05:36.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672964 2024-09-18T15:05:36.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-09-18T15:05:36.174 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672964 2024-09-18T15:05:36.174 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:05:36.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672964 -lt 42949672964 2024-09-18T15:05:36.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:05:36.478 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-68719476739 2024-09-18T15:05:36.478 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:05:36.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:05:36.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-68719476739 2024-09-18T15:05:36.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:05:36.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=68719476739 2024-09-18T15:05:36.484 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 68719476739 2024-09-18T15:05:36.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 68719476739' 2024-09-18T15:05:36.484 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:05:36.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 68719476739 -lt 68719476739 2024-09-18T15:05:36.782 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:05:36.783 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:36.783 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:37.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T15:05:37.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:37.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:37.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:37.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:37.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:37.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:37.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:37.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T15:05:37.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:37.513 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:37.513 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:37.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T15:05:37.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:05:37.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:05:37.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-09-18T15:05:37.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: create_ec_pool: local pool_name=ecpool 2024-09-18T15:05:37.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: create_ec_pool: shift 2024-09-18T15:05:37.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: create_ec_pool: local allow_overwrites=false 2024-09-18T15:05:37.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: create_ec_pool: shift 2024-09-18T15:05:37.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-18T15:05:38.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-18T15:05:38.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-18T15:05:39.588 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'ecpool' created 2024-09-18T15:05:39.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:05:40.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: create_ec_pool: '[' false = true ']' 2024-09-18T15:05:40.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: create_ec_pool: wait_for_clean 2024-09-18T15:05:40.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:05:40.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:05:40.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:05:40.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:05:40.606 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:05:40.607 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:05:40.607 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:05:40.608 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:05:40.608 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:05:40.608 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:05:40.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:05:40.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:05:40.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:05:40.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:05:40.826 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:05:41.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:05:41.290 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:05:41.290 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:05:41.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:05:41.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:05:41.290 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:05:41.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836487 2024-09-18T15:05:41.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836487 2024-09-18T15:05:41.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836487' 2024-09-18T15:05:41.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:05:41.439 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:05:41.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672966 2024-09-18T15:05:41.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672966 2024-09-18T15:05:41.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836487 1-42949672966' 2024-09-18T15:05:41.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:05:41.579 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:05:41.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=68719476741 2024-09-18T15:05:41.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 68719476741 2024-09-18T15:05:41.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836487 1-42949672966 2-68719476741' 2024-09-18T15:05:41.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:05:41.718 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836487 2024-09-18T15:05:41.718 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:05:41.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:05:41.720 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836487 2024-09-18T15:05:41.720 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:05:41.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836487 2024-09-18T15:05:41.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-09-18T15:05:41.723 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836487 2024-09-18T15:05:41.724 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:05:42.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836485 -lt 21474836487 2024-09-18T15:05:42.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:05:43.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:05:43.042 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:05:43.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836487 -lt 21474836487 2024-09-18T15:05:43.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:05:43.611 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672966 2024-09-18T15:05:43.612 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:05:43.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:05:43.615 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672966 2024-09-18T15:05:43.615 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:05:43.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672966 2024-09-18T15:05:43.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-09-18T15:05:43.617 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672966 2024-09-18T15:05:43.618 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:05:43.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672966 -lt 42949672966 2024-09-18T15:05:43.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:05:43.989 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-68719476741 2024-09-18T15:05:43.990 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:05:43.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:05:43.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-68719476741 2024-09-18T15:05:43.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:05:43.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=68719476741 2024-09-18T15:05:43.996 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 68719476741 2024-09-18T15:05:43.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 68719476741' 2024-09-18T15:05:43.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:05:44.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 68719476741 -lt 68719476741 2024-09-18T15:05:44.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:05:44.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:44.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:44.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:05:44.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:44.791 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:44.791 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:44.791 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:44.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:44.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:44.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:45.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:05:45.135 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:45.135 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:45.136 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:45.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:05:45.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:05:45.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:05:45.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: create_ec_pool: return 0 2024-09-18T15:05:45.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: auto_repair_erasure_coded: local payload=ABCDEF 2024-09-18T15:05:45.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:294: auto_repair_erasure_coded: echo ABCDEF 2024-09-18T15:05:45.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:295: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-18T15:05:45.683 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-09-18T15:05:45.683 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-09-18T15:05:45.683 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-09-18T15:05:45.684 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-09-18T15:05:45.684 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-09-18T15:05:45.684 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T15:05:45.684 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:05:45.684 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:05:46.037 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-09-18T15:05:46.037 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:05:46.037 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-09-18T15:05:46.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:05:46.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:05:46.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:05:46.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:05:46.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:05:46.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:05:46.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:05:46.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:05:46.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:05:46.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:05:46.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:05:46.336 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:05:46.336 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:05:46.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:05:46.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:05:46.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:05:46.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:05:46.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:05:46.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:05:46.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:05:46.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:05:46.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:05:46.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:05:46.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-18T15:05:47.642 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-18T15:05:47.642 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-09-18T15:05:47.643 INFO:tasks.workunit.client.0.smithi046.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-18T15:05:47.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:05:47.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:05:47.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:05:47.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:05:47.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:05:47.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:05:47.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:05:47.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:05:47.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:05:47.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:05:47.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:05:47.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:05:47.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:05:47.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:05:47.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:05:47.988 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:05:47.988 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:05:47.988 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:05:47.990 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:05:47.990 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:05:47.990 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:05:47.990 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:05:47.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:05:47.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:05:47.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:05:47.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:05:47.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:05:47.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:05:47.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:05:47.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:05:47.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:05:47.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:05:47.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:05:47.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:05:47.991 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:05:47.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:05:47.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:05:47.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:05:47.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:05:47.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:05:47.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:05:48.025 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:48.018+0000 7f5f8980c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:05:48.029 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:48.022+0000 7f5f8980c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:05:48.036 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:48.022+0000 7f5f8980c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:05:48.036 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:48.026+0000 7f5f8980c540 -1 Falling back to public interface 2024-09-18T15:05:48.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:05:48.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:05:48.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:05:48.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:05:48.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:05:48.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:05:48.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:05:48.414 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:05:48.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:05:48.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:05:48.428 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:48.422+0000 7f5f8980c540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:05:48.429 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:48.422+0000 7f5f8980c540 -1 journal do_read_entry(466944): bad header magic 2024-09-18T15:05:48.429 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:48.422+0000 7f5f8980c540 -1 journal do_read_entry(466944): bad header magic 2024-09-18T15:05:48.439 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:05:48.430+0000 7f5f8980c540 -1 osd.1 39 log_to_monitors true 2024-09-18T15:05:48.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:05:49.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:05:49.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:05:49.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:05:49.769 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:05:49.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:05:49.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:05:50.110 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 40 last_clean_interval [10,39) [v2:127.0.0.1:6810/204918,v1:127.0.0.1:6811/204918] [v2:127.0.0.1:6812/204918,v1:127.0.0.1:6813/204918] exists,up 1517c6db-ecb0-4772-9b8a-558021bd9092 2024-09-18T15:05:50.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:05:50.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:05:50.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:05:50.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:05:50.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:05:50.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:05:50.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:05:50.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:05:50.111 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:05:50.112 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:05:50.112 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:05:50.113 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:05:50.113 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:05:50.113 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:05:50.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:05:50.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:05:50.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:05:50.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:05:50.321 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:05:50.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:05:50.771 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:05:50.771 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:05:50.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:05:50.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:05:50.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:05:50.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836490 2024-09-18T15:05:50.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836490 2024-09-18T15:05:50.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836490' 2024-09-18T15:05:50.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:05:50.942 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:05:51.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=184683593730 2024-09-18T15:05:51.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 184683593730 2024-09-18T15:05:51.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836490 1-184683593730' 2024-09-18T15:05:51.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:05:51.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:05:51.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=68719476744 2024-09-18T15:05:51.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 68719476744 2024-09-18T15:05:51.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836490 1-184683593730 2-68719476744' 2024-09-18T15:05:51.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:05:51.215 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836490 2024-09-18T15:05:51.215 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:05:51.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:05:51.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836490 2024-09-18T15:05:51.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:05:51.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836490 2024-09-18T15:05:51.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-09-18T15:05:51.220 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836490 2024-09-18T15:05:51.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:05:51.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836488 -lt 21474836490 2024-09-18T15:05:51.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:05:52.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:05:52.562 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:05:52.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836490 -lt 21474836490 2024-09-18T15:05:52.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:05:52.881 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-184683593730 2024-09-18T15:05:52.881 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:05:52.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:05:52.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-184683593730 2024-09-18T15:05:52.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:05:52.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=184683593730 2024-09-18T15:05:52.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 184683593730' 2024-09-18T15:05:52.887 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 184683593730 2024-09-18T15:05:52.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:05:53.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 184683593730 -lt 184683593730 2024-09-18T15:05:53.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:05:53.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-68719476744 2024-09-18T15:05:53.194 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:05:53.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:05:53.197 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-68719476744 2024-09-18T15:05:53.200 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:05:53.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=68719476744 2024-09-18T15:05:53.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 68719476744' 2024-09-18T15:05:53.201 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 68719476744 2024-09-18T15:05:53.201 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:05:53.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 68719476744 -lt 68719476744 2024-09-18T15:05:53.565 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:05:53.566 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:53.566 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:54.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:05:54.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:54.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:54.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:54.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:54.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:54.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:54.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:54.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:05:54.306 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:54.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:54.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:54.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:05:54.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:05:54.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:05:54.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:301: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-09-18T15:05:54.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-09-18T15:05:54.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T15:05:54.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:05:54.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:05:55.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:301: auto_repair_erasure_coded: local pgid=2.0 2024-09-18T15:05:55.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-09-18T15:05:55.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:05:55.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:05:55.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:05:55.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:05:55.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-09-18T15:05:45.918746+0000 2024-09-18T15:05:55.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=2.0 2024-09-18T15:05:55.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:05:45.918746+0000 2024-09-18T15:05:55.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:05:55.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:05:55.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:05:55.390 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:05:55.390 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:05:55.390 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:05:55.391 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:05:55.391 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:05:55.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:05:45.918746+0000 '>' 2024-09-18T15:05:45.918746+0000 2024-09-18T15:05:55.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:05:56.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:05:56.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:05:56.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:05:56.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:05:56.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:05:56.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:05:56.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:05:57.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:05:45.918746+0000 '>' 2024-09-18T15:05:45.918746+0000 2024-09-18T15:05:57.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:05:58.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:05:58.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:05:58.067 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:05:58.067 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:05:58.067 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:05:58.068 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:05:58.068 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:05:58.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:05:45.918746+0000 '>' 2024-09-18T15:05:45.918746+0000 2024-09-18T15:05:58.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:05:59.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:05:59.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:05:59.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:05:59.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:05:59.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:05:59.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:05:59.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:05:59.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:05:45.918746+0000 '>' 2024-09-18T15:05:45.918746+0000 2024-09-18T15:05:59.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:06:00.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:06:00.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:06:00.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:06:00.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:06:00.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:06:00.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:06:00.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:06:01.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:05:54.976074+0000 '>' 2024-09-18T15:05:45.918746+0000 2024-09-18T15:06:01.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:06:01.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: wait_for_clean 2024-09-18T15:06:01.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:06:01.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:06:01.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:06:01.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:06:01.047 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:06:01.048 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:06:01.048 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:06:01.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:06:01.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:06:01.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:06:01.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:06:01.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:06:01.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:06:01.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:06:01.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:06:01.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:06:01.579 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:06:01.579 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:06:01.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:06:01.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:06:01.579 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:06:01.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836493 2024-09-18T15:06:01.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836493 2024-09-18T15:06:01.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836493' 2024-09-18T15:06:01.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:06:01.754 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:06:01.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=184683593733 2024-09-18T15:06:01.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 184683593733 2024-09-18T15:06:01.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836493 1-184683593733' 2024-09-18T15:06:01.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:06:01.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:06:02.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=68719476747 2024-09-18T15:06:02.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 68719476747 2024-09-18T15:06:02.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836493 1-184683593733 2-68719476747' 2024-09-18T15:06:02.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:06:02.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836493 2024-09-18T15:06:02.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:06:02.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:06:02.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836493 2024-09-18T15:06:02.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:06:02.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836493 2024-09-18T15:06:02.075 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836493 2024-09-18T15:06:02.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-09-18T15:06:02.076 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:06:02.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836493 -lt 21474836493 2024-09-18T15:06:02.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:06:02.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-184683593733 2024-09-18T15:06:02.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:06:02.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:06:02.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-184683593733 2024-09-18T15:06:02.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:06:02.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=184683593733 2024-09-18T15:06:02.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 184683593733' 2024-09-18T15:06:02.388 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 184683593733 2024-09-18T15:06:02.389 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:06:02.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 184683593733 -lt 184683593733 2024-09-18T15:06:02.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:06:02.753 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-68719476747 2024-09-18T15:06:02.753 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:06:02.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:06:02.755 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:06:02.755 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-68719476747 2024-09-18T15:06:02.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=68719476747 2024-09-18T15:06:02.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 68719476747' 2024-09-18T15:06:02.756 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 68719476747 2024-09-18T15:06:02.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:06:03.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 68719476746 -lt 68719476747 2024-09-18T15:06:03.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:06:04.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:06:04.113 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:06:04.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 68719476747 -lt 68719476747 2024-09-18T15:06:04.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:06:04.436 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:04.436 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:04.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:06:04.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:04.804 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:04.804 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:04.804 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:04.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:04.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:04.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:05.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:06:05.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:05.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:05.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:05.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:06:05.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:06:05.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:06:05.531 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:306: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-09-18T15:06:05.531 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-09-18T15:06:05.531 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-09-18T15:06:05.532 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-09-18T15:06:05.532 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-09-18T15:06:05.532 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T15:06:05.532 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:06:05.533 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:06:05.836 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-09-18T15:06:05.837 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:06:05.837 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-09-18T15:06:06.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:306: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:06:06.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:06:06.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:06:06.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:06:06.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:06:06.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:06:06.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:06:06.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:06:06.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:06:06.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:06:06.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:06:06.141 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:06:06.142 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:06:06.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:06:06.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:06:06.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:06:06.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:06:06.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:06:06.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:06:06.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:06:06.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:06:06.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:06:06.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:06:06.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-18T15:06:07.351 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-18T15:06:07.352 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-09-18T15:06:07.352 INFO:tasks.workunit.client.0.smithi046.stdout:_ 2024-09-18T15:06:07.352 INFO:tasks.workunit.client.0.smithi046.stdout:hinfo_key 2024-09-18T15:06:07.352 INFO:tasks.workunit.client.0.smithi046.stdout:snapset 2024-09-18T15:06:07.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:06:07.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:06:07.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:06:07.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:06:07.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:06:07.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:06:07.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:06:07.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:06:07.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:06:07.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:06:07.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:06:07.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:06:07.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:06:07.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:06:07.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:06:07.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:06:07.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:06:07.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:06:07.852 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:06:07.852 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:06:07.852 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:06:07.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:06:07.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:06:07.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:06:07.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:06:07.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:06:07.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:06:07.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:06:07.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:06:07.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:06:07.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:06:07.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:06:07.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:06:07.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:06:07.853 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:06:07.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:06:07.855 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:06:07.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:06:07.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:06:07.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:06:07.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:06:07.891 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:07.881+0000 7fc0e46c6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:06:07.928 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:07.917+0000 7fc0e46c6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:06:07.929 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:07.921+0000 7fc0e46c6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:06:07.932 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:07.921+0000 7fc0e46c6540 -1 Falling back to public interface 2024-09-18T15:06:08.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:06:08.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:06:08.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:06:08.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:06:08.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:06:08.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:06:08.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:06:08.263 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:06:08.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:06:08.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:06:08.607 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:08.597+0000 7fc0e46c6540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:06:08.622 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:08.613+0000 7fc0e46c6540 -1 journal do_read_entry(806912): bad header magic 2024-09-18T15:06:08.622 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:08.613+0000 7fc0e46c6540 -1 journal do_read_entry(806912): bad header magic 2024-09-18T15:06:08.647 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:08.637+0000 7fc0e46c6540 -1 osd.1 53 log_to_monitors true 2024-09-18T15:06:08.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:06:09.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:06:09.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:06:09.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:06:09.681 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:06:09.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:06:09.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:06:10.343 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 59 up_thru 59 down_at 54 last_clean_interval [43,53) [v2:127.0.0.1:6810/206135,v1:127.0.0.1:6811/206135] [v2:127.0.0.1:6812/206135,v1:127.0.0.1:6813/206135] exists,up 1517c6db-ecb0-4772-9b8a-558021bd9092 2024-09-18T15:06:10.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:06:10.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:06:10.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:06:10.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:06:10.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:06:10.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:06:10.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:06:10.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:06:10.344 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:06:10.345 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:06:10.345 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:06:10.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:06:10.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:06:10.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:06:10.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:06:10.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:06:10.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:06:10.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:06:10.560 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:06:10.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:06:10.873 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:06:10.873 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:06:10.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:06:10.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:06:10.874 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:06:11.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836496 2024-09-18T15:06:11.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836496 2024-09-18T15:06:11.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836496' 2024-09-18T15:06:11.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:06:11.076 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:06:11.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=253403070466 2024-09-18T15:06:11.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 253403070466 2024-09-18T15:06:11.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836496 1-253403070466' 2024-09-18T15:06:11.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:06:11.222 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:06:11.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=68719476750 2024-09-18T15:06:11.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 68719476750 2024-09-18T15:06:11.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836496 1-253403070466 2-68719476750' 2024-09-18T15:06:11.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:06:11.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836496 2024-09-18T15:06:11.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:06:11.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:06:11.366 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836496 2024-09-18T15:06:11.366 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:06:11.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836496 2024-09-18T15:06:11.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-09-18T15:06:11.368 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836496 2024-09-18T15:06:11.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:06:11.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836494 -lt 21474836496 2024-09-18T15:06:11.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:06:12.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:06:12.678 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:06:12.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836496 -lt 21474836496 2024-09-18T15:06:12.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:06:13.000 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-253403070466 2024-09-18T15:06:13.000 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:06:13.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:06:13.003 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-253403070466 2024-09-18T15:06:13.004 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:06:13.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=253403070466 2024-09-18T15:06:13.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 253403070466' 2024-09-18T15:06:13.005 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 253403070466 2024-09-18T15:06:13.006 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:06:13.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 253403070466 -lt 253403070466 2024-09-18T15:06:13.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:06:13.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-68719476750 2024-09-18T15:06:13.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:06:13.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:06:13.310 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-68719476750 2024-09-18T15:06:13.310 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:06:13.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=68719476750 2024-09-18T15:06:13.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 68719476750' 2024-09-18T15:06:13.312 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 68719476750 2024-09-18T15:06:13.313 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:06:13.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 68719476750 -lt 68719476750 2024-09-18T15:06:13.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:06:13.659 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:13.659 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:14.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:06:14.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:14.258 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:14.258 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:14.258 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:14.258 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:14.259 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:14.259 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:14.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:06:14.563 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:14.564 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:14.564 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:14.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:06:14.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:06:14.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:06:14.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:307: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-18T15:06:15.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:308: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-18T15:06:15.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T15:06:15.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:06:15.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:06:15.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:06:15.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:06:15.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:06:15.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:06:15.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:06:15.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:06:15.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:06:15.204 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:06:15.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:06:15.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:06:15.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:06:15.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:06:15.209 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:06:15.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:06:15.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:06:15.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:06:15.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:06:15.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:06:15.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:06:15.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:06:15.240 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:06:15.240 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:06:15.240 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:06:15.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:06:15.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:06:15.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:06:15.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T15:06:15.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T15:06:15.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T15:06:15.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T15:06:15.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:06:15.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:06:15.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:06:15.243 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:06:15.243 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:06:15.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:06:15.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:06:15.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:06:15.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:06:15.248 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:06:15.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:06:15.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:06:15.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:06:15.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:06:15.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:06:15.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:06:15.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:06:15.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:06:15.255 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:06:15.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:06:15.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:06:15.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:06:15.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:06:15.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:06:15.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:06:15.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:06:15.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:06:15.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:06:15.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T15:06:15.265 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:06:15.265 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:06:15.266 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:06:15.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T15:06:15.268 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:06:15.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:06:15.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:06:15.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T15:06:15.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_auto_repair_erasure_coded_overwrites td/osd-scrub-repair 2024-09-18T15:06:15.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:316: TEST_auto_repair_erasure_coded_overwrites: '[' true = true ']' 2024-09-18T15:06:15.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:317: TEST_auto_repair_erasure_coded_overwrites: auto_repair_erasure_coded td/osd-scrub-repair true 2024-09-18T15:06:15.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:267: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-18T15:06:15.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local allow_overwrites=true 2024-09-18T15:06:15.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local poolname=ecpool 2024-09-18T15:06:15.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-09-18T15:06:15.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T15:06:15.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:06:15.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:06:15.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:06:15.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T15:06:15.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-18T15:06:15.657 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:06:15.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:06:15.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:06:15.658 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:06:15.658 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:06:15.658 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:06:15.659 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:06:15.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T15:06:15.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:06:15.974 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:06:15.974 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:06:15.974 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:06:15.974 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:06:15.974 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:06:15.975 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:06:15.975 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:06:15.975 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:06:15.977 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:06:15.977 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:06:15.977 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:06:15.977 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:06:15.977 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:06:15.978 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T15:06:16.069 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:06:16.069 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:06:16.069 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:06:16.069 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:06:16.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:06:16.071 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:06:16.071 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:06:16.071 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:06:16.071 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:06:16.071 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:06:16.071 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:06:16.071 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:06:16.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:06:16.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T15:06:16.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-09-18T15:06:16.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T15:06:16.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:06:16.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:06:16.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:06:16.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T15:06:16.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:06:17.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:06:17.023 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:06:17.023 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:06:17.023 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:06:17.023 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:06:17.024 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:06:17.024 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:06:17.025 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:06:17.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:06:17.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:278: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:06:17.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: seq 0 2 2024-09-18T15:06:17.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-18T15:06:17.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: '[' true = true ']' 2024-09-18T15:06:17.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:06:17.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:06:17.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:06:17.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:06:17.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:06:17.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:06:17.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:06:17.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:06:17.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:06:17.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:06:17.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:06:17.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:06:17.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:06:17.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:06:17.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:06:17.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:06:17.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:06:17.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:06:17.070 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:06:17.070 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:06:17.070 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:06:17.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:06:17.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:06:17.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:06:17.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:06:17.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:06:17.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:06:17.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:06:17.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:06:17.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:06:17.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:06:17.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:06:17.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:06:17.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:06:17.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:06:17.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:06:17.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ead7a52a-5233-40dd-9910-6c6f79bfbf49 2024-09-18T15:06:17.073 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 ead7a52a-5233-40dd-9910-6c6f79bfbf49 2024-09-18T15:06:17.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ead7a52a-5233-40dd-9910-6c6f79bfbf49' 2024-09-18T15:06:17.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:06:17.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBp7OpmuP1CBRAAq3kCTD92YrsviIR5j7H0tQ== 2024-09-18T15:06:17.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBp7OpmuP1CBRAAq3kCTD92YrsviIR5j7H0tQ=="}' 2024-09-18T15:06:17.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ead7a52a-5233-40dd-9910-6c6f79bfbf49 -i td/osd-scrub-repair/0/new.json 2024-09-18T15:06:17.365 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:06:17.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-18T15:06:17.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBp7OpmuP1CBRAAq3kCTD92YrsviIR5j7H0tQ== --osd-uuid ead7a52a-5233-40dd-9910-6c6f79bfbf49 2024-09-18T15:06:17.409 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:17.401+0000 7fe63b8df540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:06:17.412 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:17.405+0000 7fe63b8df540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:06:17.414 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:17.405+0000 7fe63b8df540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:06:17.415 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:17.405+0000 7fe63b8df540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:06:17.415 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:17.405+0000 7fe63b8df540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:06:17.415 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:17.405+0000 7fe63b8df540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:06:17.489 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:17.481+0000 7fe63b8df540 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-18T15:06:23.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T15:06:23.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:06:23.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:06:23.644 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T15:06:23.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:06:24.029 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T15:06:24.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:06:24.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:06:24.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:06:24.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:06:24.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:06:24.063 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:24.053+0000 7fcc54daa540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:06:24.065 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:24.057+0000 7fcc54daa540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:06:24.066 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:24.057+0000 7fcc54daa540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:06:24.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:06:24.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:06:24.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:06:24.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:06:24.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:06:24.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:06:24.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:06:24.311 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:06:24.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:06:24.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:06:24.607 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:24.597+0000 7fcc54daa540 -1 Falling back to public interface 2024-09-18T15:06:24.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:06:25.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:06:25.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:06:25.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:06:25.669 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:06:25.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:06:25.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:06:25.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:06:26.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:06:26.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:06:26.975 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:06:26.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:06:26.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:06:26.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:06:27.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:06:27.298 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:27.289+0000 7fcc54daa540 -1 osd.0 0 log_to_monitors true 2024-09-18T15:06:28.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:06:28.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:06:28.291 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:06:28.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:06:28.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:06:28.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:06:28.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:06:29.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:06:29.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:06:29.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:06:29.603 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:06:29.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:06:29.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:06:29.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:06:30.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:06:30.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:06:30.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:06:30.922 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:06:30.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:06:30.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:06:31.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:06:32.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:06:32.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:06:32.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:06:32.331 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:06:32.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:06:32.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:06:32.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:06:33.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:06:33.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:06:33.651 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:06:33.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:06:33.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:06:33.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:06:34.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:06:35.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:06:35.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:06:35.070 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:06:35.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:06:35.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:06:35.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:06:35.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:06:36.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:06:36.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:06:36.393 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:06:36.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:06:36.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:06:36.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:06:36.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:06:37.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:06:37.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:06:37.716 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:06:37.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:06:37.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:06:37.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:06:38.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:06:39.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:06:39.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:06:39.088 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:06:39.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:06:39.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:06:39.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:06:39.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:06:40.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:06:40.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:06:40.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:06:40.409 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:06:40.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:06:40.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:06:41.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:06:42.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:06:42.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:06:42.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:06:42.138 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:06:42.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:06:42.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:06:42.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:06:43.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:06:43.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:06:43.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:06:43.695 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:06:43.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:06:43.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:06:44.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:06:45.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:06:45.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:06:45.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:06:45.103 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:06:45.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:06:45.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:06:45.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:06:46.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:06:46.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:06:46.669 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:06:46.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:06:46.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:06:46.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:06:47.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:06:48.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:06:48.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:06:48.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:06:48.314 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:06:48.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:06:48.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:06:48.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:06:49.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:06:49.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:06:49.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:06:49.957 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:06:49.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:06:49.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:06:50.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:06:51.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:06:51.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:06:51.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:06:51.384 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:06:51.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:06:51.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:06:51.701 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/207836,v1:127.0.0.1:6803/207836] [v2:127.0.0.1:6804/207836,v1:127.0.0.1:6805/207836] exists,up ead7a52a-5233-40dd-9910-6c6f79bfbf49 2024-09-18T15:06:51.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:06:51.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:06:51.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:06:51.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-18T15:06:51.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: '[' true = true ']' 2024-09-18T15:06:51.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:06:51.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:06:51.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:06:51.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:06:51.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:06:51.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:06:51.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:06:51.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:06:51.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:06:51.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:06:51.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:06:51.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:06:51.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:06:51.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:06:51.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:06:51.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:06:51.707 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:06:51.707 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:06:51.707 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:06:51.707 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:06:51.707 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:06:51.707 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:06:51.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:06:51.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:06:51.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:06:51.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:06:51.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:06:51.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:06:51.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:06:51.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:06:51.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:06:51.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:06:51.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:06:51.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:06:51.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:06:51.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:06:51.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=786eb5db-6fdf-4b87-87ac-79f18adc37d3 2024-09-18T15:06:51.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 786eb5db-6fdf-4b87-87ac-79f18adc37d3' 2024-09-18T15:06:51.710 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 786eb5db-6fdf-4b87-87ac-79f18adc37d3 2024-09-18T15:06:51.710 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:06:51.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCL7OpmkTIYKxAAZtkh5OGgzt0Nhk6w0EiqtA== 2024-09-18T15:06:51.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCL7OpmkTIYKxAAZtkh5OGgzt0Nhk6w0EiqtA=="}' 2024-09-18T15:06:51.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 786eb5db-6fdf-4b87-87ac-79f18adc37d3 -i td/osd-scrub-repair/1/new.json 2024-09-18T15:06:52.128 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:06:52.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-18T15:06:52.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCL7OpmkTIYKxAAZtkh5OGgzt0Nhk6w0EiqtA== --osd-uuid 786eb5db-6fdf-4b87-87ac-79f18adc37d3 2024-09-18T15:06:52.183 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:52.173+0000 7fb0e4d63540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:06:52.187 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:52.177+0000 7fb0e4d63540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:06:52.188 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:52.177+0000 7fb0e4d63540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:06:52.189 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:52.181+0000 7fb0e4d63540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:06:52.189 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:52.181+0000 7fb0e4d63540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:06:52.189 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:52.181+0000 7fb0e4d63540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:06:52.293 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:52.285+0000 7fb0e4d63540 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-18T15:06:58.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-18T15:06:58.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:06:58.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:06:58.596 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T15:06:58.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:06:58.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:06:58.979 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T15:06:58.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:06:58.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:06:58.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:06:58.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:06:59.014 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:59.005+0000 7fb756af5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:06:59.016 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:59.009+0000 7fb756af5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:06:59.018 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:59.009+0000 7fb756af5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:06:59.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:06:59.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:06:59.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:06:59.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:06:59.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:06:59.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:06:59.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:06:59.334 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:06:59.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:06:59.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:06:59.574 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:06:59.565+0000 7fb756af5540 -1 Falling back to public interface 2024-09-18T15:06:59.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:00.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:00.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:00.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:07:00.645 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:07:00.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:00.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:07:00.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:01.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:01.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:01.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:07:01.958 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:07:01.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:01.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:07:02.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:02.414 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:07:02.405+0000 7fb756af5540 -1 osd.1 0 log_to_monitors true 2024-09-18T15:07:03.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:03.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:03.272 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:07:03.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:07:03.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:03.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:07:03.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:04.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:04.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:04.670 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:07:04.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:07:04.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:04.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:07:04.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:05.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:05.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:05.994 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:07:05.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:07:05.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:05.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:07:06.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:07.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:07.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:07.334 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:07:07.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:07:07.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:07.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:07:07.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:08.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:08.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:08.694 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:07:08.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:07:08.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:08.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:07:09.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:10.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:10.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:10.019 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:07:10.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:07:10.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:10.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:07:10.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:11.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:11.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:11.333 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:07:11.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:07:11.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:11.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:07:11.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:12.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:12.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:12.755 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:07:12.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:07:12.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:12.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:07:13.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:14.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:14.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:14.116 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:07:14.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:07:14.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:14.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:07:14.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:15.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:15.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:15.531 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:07:15.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:07:15.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:15.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:07:15.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:16.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:16.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:16.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:07:16.846 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:07:16.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:16.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:07:17.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:18.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:18.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:18.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:07:18.469 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:07:18.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:18.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:07:19.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:20.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:20.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:20.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:07:20.279 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:07:20.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:20.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:07:21.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:22.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:22.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:22.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:07:22.063 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:07:22.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:22.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:07:22.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:23.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:23.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:23.541 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:07:23.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:07:23.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:23.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:07:24.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:25.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:25.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:25.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:07:25.309 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:07:25.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:25.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:07:26.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:27.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:27.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:27.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:07:27.061 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:07:27.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:27.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:07:27.379 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/209651,v1:127.0.0.1:6811/209651] [v2:127.0.0.1:6812/209651,v1:127.0.0.1:6813/209651] exists,up 786eb5db-6fdf-4b87-87ac-79f18adc37d3 2024-09-18T15:07:27.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:07:27.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:07:27.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:07:27.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-18T15:07:27.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: '[' true = true ']' 2024-09-18T15:07:27.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:07:27.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:07:27.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:07:27.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T15:07:27.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:07:27.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:07:27.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:07:27.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:07:27.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:07:27.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:07:27.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T15:07:27.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T15:07:27.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:07:27.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:07:27.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:07:27.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:07:27.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:07:27.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:07:27.385 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:07:27.385 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:07:27.385 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:07:27.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:07:27.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:07:27.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:07:27.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:07:27.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:07:27.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:07:27.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:07:27.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:07:27.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:07:27.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:07:27.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:07:27.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:07:27.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:07:27.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-18T15:07:27.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:07:27.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=528f8230-378f-48b8-b50c-71276b4c51ba 2024-09-18T15:07:27.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 528f8230-378f-48b8-b50c-71276b4c51ba' 2024-09-18T15:07:27.388 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 528f8230-378f-48b8-b50c-71276b4c51ba 2024-09-18T15:07:27.388 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:07:27.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCv7Opmv6MvGBAAN+P6oPuBKMHRrtxhGErISQ== 2024-09-18T15:07:27.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCv7Opmv6MvGBAAN+P6oPuBKMHRrtxhGErISQ=="}' 2024-09-18T15:07:27.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 528f8230-378f-48b8-b50c-71276b4c51ba -i td/osd-scrub-repair/2/new.json 2024-09-18T15:07:27.797 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:07:27.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-18T15:07:27.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCv7Opmv6MvGBAAN+P6oPuBKMHRrtxhGErISQ== --osd-uuid 528f8230-378f-48b8-b50c-71276b4c51ba 2024-09-18T15:07:27.848 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:07:27.836+0000 7f136f606540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:07:27.862 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:07:27.852+0000 7f136f606540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:07:27.864 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:07:27.852+0000 7f136f606540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:07:27.865 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:07:27.856+0000 7f136f606540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T15:07:27.865 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:07:27.856+0000 7f136f606540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T15:07:27.865 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:07:27.856+0000 7f136f606540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T15:07:27.934 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:07:27.924+0000 7f136f606540 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-18T15:07:34.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-18T15:07:34.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:07:34.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T15:07:34.964 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T15:07:34.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:07:35.350 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T15:07:35.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T15:07:35.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:07:35.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:07:35.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:07:35.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:07:35.386 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:07:35.376+0000 7fb84fc32540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:07:35.415 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:07:35.404+0000 7fb84fc32540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:07:35.416 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:07:35.408+0000 7fb84fc32540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:07:35.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T15:07:35.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:07:35.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:07:35.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:07:35.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:07:35.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:35.714 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:07:35.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:07:35.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:35.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:07:35.959 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:07:35.948+0000 7fb84fc32540 -1 Falling back to public interface 2024-09-18T15:07:36.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:37.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:37.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:37.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:07:37.072 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:07:37.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:37.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:07:37.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:38.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:38.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:38.442 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:07:38.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:07:38.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:38.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:07:38.723 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:07:38.712+0000 7fb84fc32540 -1 osd.2 0 log_to_monitors true 2024-09-18T15:07:38.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:39.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:39.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:39.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:07:39.760 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:07:39.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:39.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:07:40.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:41.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:41.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:41.165 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:07:41.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:07:41.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:41.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:07:41.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:42.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:42.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:42.766 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:07:42.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:07:42.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:42.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:07:43.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:44.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:44.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:44.149 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:07:44.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:07:44.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:44.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:07:44.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:45.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:45.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:45.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:07:45.465 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:07:45.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:45.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:07:45.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:46.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:46.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:46.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:07:46.959 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:07:46.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:07:46.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:47.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:48.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:48.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:48.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:07:48.307 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:07:48.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:48.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:07:48.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:49.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:49.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:49.759 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:07:49.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:07:49.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:49.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:07:50.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:51.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:51.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:51.338 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:07:51.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:07:51.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:51.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:07:51.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:52.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:52.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:52.493 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:07:52.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:07:52.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:52.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:07:52.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:53.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:53.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:53.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:07:53.810 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:07:53.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:53.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:07:54.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:55.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:55.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:55.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:07:55.364 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:07:55.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:55.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:07:56.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:57.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:57.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:57.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:07:57.038 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:07:57.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:57.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:07:57.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:07:58.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:07:58.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:07:58.722 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:07:58.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:07:58.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:07:58.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:07:59.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:08:00.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:08:00.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:08:00.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:08:00.621 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:08:00.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:08:00.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:08:01.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:08:02.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:08:02.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:08:02.399 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:08:02.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:08:02.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:08:02.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:08:03.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:08:04.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:08:04.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:08:04.091 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:08:04.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:08:04.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:08:04.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:08:04.595 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 34 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/211456,v1:127.0.0.1:6819/211456] [v2:127.0.0.1:6820/211456,v1:127.0.0.1:6821/211456] exists,up 528f8230-378f-48b8-b50c-71276b4c51ba 2024-09-18T15:08:04.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:08:04.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:08:04.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:08:04.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: auto_repair_erasure_coded: create_rbd_pool 2024-09-18T15:08:04.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T15:08:05.007 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' does not exist 2024-09-18T15:08:05.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T15:08:05.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T15:08:05.570 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' created 2024-09-18T15:08:05.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:08:06.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T15:08:08.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: wait_for_clean 2024-09-18T15:08:08.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:08:08.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:08:08.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:08:08.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:08:08.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:08:08.702 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:08:08.702 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:08:08.703 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:08:08.703 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:08:08.703 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:08:08.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:08:08.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:08:08.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:08:08.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:08:08.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:08:09.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:08:09.225 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:08:09.226 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:08:09.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:08:09.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:08:09.226 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:08:09.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836504 2024-09-18T15:08:09.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836504 2024-09-18T15:08:09.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836504' 2024-09-18T15:08:09.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:08:09.364 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:08:09.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=68719476752 2024-09-18T15:08:09.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 68719476752 2024-09-18T15:08:09.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836504 1-68719476752' 2024-09-18T15:08:09.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:08:09.503 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:08:09.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=146028888069 2024-09-18T15:08:09.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 146028888069 2024-09-18T15:08:09.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836504 1-68719476752 2-146028888069' 2024-09-18T15:08:09.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:08:09.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836504 2024-09-18T15:08:09.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:08:09.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:08:09.644 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836504 2024-09-18T15:08:09.644 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:08:09.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836504 2024-09-18T15:08:09.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-18T15:08:09.647 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836504 2024-09-18T15:08:09.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:08:10.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836503 -lt 21474836504 2024-09-18T15:08:10.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:08:11.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:08:11.042 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:08:11.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836504 -lt 21474836504 2024-09-18T15:08:11.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:08:11.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-68719476752 2024-09-18T15:08:11.436 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:08:11.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:08:11.439 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-68719476752 2024-09-18T15:08:11.439 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:08:11.440 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 68719476752 2024-09-18T15:08:11.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=68719476752 2024-09-18T15:08:11.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 68719476752' 2024-09-18T15:08:11.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:08:11.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 68719476752 -lt 68719476752 2024-09-18T15:08:11.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:08:11.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-146028888069 2024-09-18T15:08:11.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:08:11.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:08:11.781 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-146028888069 2024-09-18T15:08:11.781 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:08:11.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=146028888069 2024-09-18T15:08:11.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 146028888069' 2024-09-18T15:08:11.783 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 146028888069 2024-09-18T15:08:11.784 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:08:12.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 146028888070 -lt 146028888069 2024-09-18T15:08:12.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:08:12.083 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:08:12.083 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:08:12.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T15:08:12.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:08:12.447 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:08:12.447 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:08:12.447 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:08:12.448 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:08:12.448 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:08:12.448 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:08:12.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T15:08:12.808 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:08:12.809 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:08:12.809 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:08:13.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T15:08:13.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:08:13.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:08:13.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-09-18T15:08:13.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: create_ec_pool: local pool_name=ecpool 2024-09-18T15:08:13.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: create_ec_pool: shift 2024-09-18T15:08:13.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: create_ec_pool: local allow_overwrites=true 2024-09-18T15:08:13.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: create_ec_pool: shift 2024-09-18T15:08:13.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-18T15:08:13.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-18T15:08:13.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-18T15:08:14.867 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'ecpool' created 2024-09-18T15:08:14.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:08:15.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: create_ec_pool: '[' true = true ']' 2024-09-18T15:08:15.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-18T15:08:16.607 INFO:tasks.workunit.client.0.smithi046.stderr:set pool 2 allow_ec_overwrites to true 2024-09-18T15:08:16.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: create_ec_pool: wait_for_clean 2024-09-18T15:08:16.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:08:16.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:08:16.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:08:16.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:08:16.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:08:16.622 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:08:16.622 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:08:16.622 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:08:16.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:08:16.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:08:16.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:08:16.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:08:16.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:08:16.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:08:16.837 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:08:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:08:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:08:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:08:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:08:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:08:17.256 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:08:17.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836507 2024-09-18T15:08:17.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836507 2024-09-18T15:08:17.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507' 2024-09-18T15:08:17.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:08:17.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:08:17.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=68719476754 2024-09-18T15:08:17.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 68719476754 2024-09-18T15:08:17.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507 1-68719476754' 2024-09-18T15:08:17.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:08:17.540 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:08:17.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=146028888072 2024-09-18T15:08:17.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 146028888072 2024-09-18T15:08:17.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507 1-68719476754 2-146028888072' 2024-09-18T15:08:17.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:08:17.682 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836507 2024-09-18T15:08:17.683 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:08:17.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:08:17.685 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836507 2024-09-18T15:08:17.686 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:08:17.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836507 2024-09-18T15:08:17.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-18T15:08:17.687 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836507 2024-09-18T15:08:17.688 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:08:18.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836505 -lt 21474836507 2024-09-18T15:08:18.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:08:19.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:08:19.006 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:08:19.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836507 -lt 21474836507 2024-09-18T15:08:19.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:08:19.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-68719476754 2024-09-18T15:08:19.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:08:19.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:08:19.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-68719476754 2024-09-18T15:08:19.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:08:19.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=68719476754 2024-09-18T15:08:19.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 68719476754' 2024-09-18T15:08:19.363 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 68719476754 2024-09-18T15:08:19.364 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:08:19.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 68719476755 -lt 68719476754 2024-09-18T15:08:19.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:08:19.682 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-146028888072 2024-09-18T15:08:19.683 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:08:19.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:08:19.685 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-146028888072 2024-09-18T15:08:19.685 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:08:19.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=146028888072 2024-09-18T15:08:19.687 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 146028888072 2024-09-18T15:08:19.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 146028888072' 2024-09-18T15:08:19.688 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:08:19.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 146028888072 -lt 146028888072 2024-09-18T15:08:19.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:08:19.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:08:19.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:08:20.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:08:20.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:08:20.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:08:20.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:08:20.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:08:20.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:08:20.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:08:20.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:08:20.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:08:20.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:08:20.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:08:20.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:08:21.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:08:21.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:08:21.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:08:21.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: create_ec_pool: return 0 2024-09-18T15:08:21.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: auto_repair_erasure_coded: local payload=ABCDEF 2024-09-18T15:08:21.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:294: auto_repair_erasure_coded: echo ABCDEF 2024-09-18T15:08:21.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:295: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-18T15:08:21.428 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-09-18T15:08:21.428 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-09-18T15:08:21.428 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-09-18T15:08:21.429 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-09-18T15:08:21.429 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-09-18T15:08:21.429 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T15:08:21.429 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:08:21.429 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:08:21.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-09-18T15:08:21.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-09-18T15:08:21.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:08:22.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:08:22.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:08:22.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:08:22.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:08:22.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:08:22.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:08:22.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:08:22.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:08:22.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:08:22.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:08:22.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:08:22.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:08:22.087 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:08:22.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:08:22.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:08:22.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:08:23.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:08:23.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:08:23.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:08:23.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:08:23.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:08:23.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:08:23.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:08:23.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-18T15:08:24.928 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-18T15:08:24.928 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-09-18T15:08:24.928 INFO:tasks.workunit.client.0.smithi046.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-18T15:08:25.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:08:25.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:08:25.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:08:25.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:08:25.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:08:25.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:08:25.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:08:25.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:08:25.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:08:25.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:08:25.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:08:25.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:08:25.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:08:25.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:08:25.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:08:25.464 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:08:25.465 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:08:25.465 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:08:25.466 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:08:25.466 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:08:25.466 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:08:25.467 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:08:25.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:08:25.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:08:25.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:08:25.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:08:25.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:08:25.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:08:25.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:08:25.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:08:25.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:08:25.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:08:25.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:08:25.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:08:25.468 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:08:25.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:08:25.469 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:08:25.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:08:25.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:08:25.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:08:25.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:08:25.505 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:08:25.500+0000 7f3612b87540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:08:25.532 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:08:25.524+0000 7f3612b87540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:08:25.533 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:08:25.528+0000 7f3612b87540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:08:25.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:08:25.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:08:25.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:08:25.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:08:25.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:08:25.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:08:25.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:08:25.805 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:08:25.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:08:25.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:08:26.079 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:08:26.072+0000 7f3612b87540 -1 Falling back to public interface 2024-09-18T15:08:26.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:08:27.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:08:27.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:08:27.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:08:27.120 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:08:27.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:08:27.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:08:27.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:08:28.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:08:28.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:08:28.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:08:28.445 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:08:28.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:08:28.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:08:28.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:08:28.936 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:08:28.928+0000 7f3612b87540 -1 osd.1 58 log_to_monitors true 2024-09-18T15:08:29.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:08:29.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:08:29.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:08:29.763 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:08:29.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:08:29.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:08:30.146 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 64 up_thru 64 down_at 59 last_clean_interval [16,58) [v2:127.0.0.1:6810/213799,v1:127.0.0.1:6811/213799] [v2:127.0.0.1:6812/213799,v1:127.0.0.1:6813/213799] exists,up 786eb5db-6fdf-4b87-87ac-79f18adc37d3 2024-09-18T15:08:30.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:08:30.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:08:30.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:08:30.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:08:30.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:08:30.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:08:30.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:08:30.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:08:30.147 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:08:30.148 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:08:30.148 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:08:30.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:08:30.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:08:30.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:08:30.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:08:30.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:08:30.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:08:30.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:08:30.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:08:30.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:08:30.791 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:08:30.791 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:08:30.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:08:30.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:08:30.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:08:30.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836510 2024-09-18T15:08:30.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836510 2024-09-18T15:08:30.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836510' 2024-09-18T15:08:30.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:08:30.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:08:31.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=274877906946 2024-09-18T15:08:31.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 274877906946 2024-09-18T15:08:31.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836510 1-274877906946' 2024-09-18T15:08:31.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:08:31.232 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:08:31.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=146028888076 2024-09-18T15:08:31.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 146028888076 2024-09-18T15:08:31.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836510 1-274877906946 2-146028888076' 2024-09-18T15:08:31.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:08:31.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836510 2024-09-18T15:08:31.426 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:08:31.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:08:31.429 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836510 2024-09-18T15:08:31.429 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:08:31.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836510 2024-09-18T15:08:31.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-09-18T15:08:31.432 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836510 2024-09-18T15:08:31.432 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:08:31.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836509 -lt 21474836510 2024-09-18T15:08:31.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:08:32.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:08:32.754 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:08:33.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836511 -lt 21474836510 2024-09-18T15:08:33.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:08:33.099 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-274877906946 2024-09-18T15:08:33.099 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:08:33.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:08:33.103 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-274877906946 2024-09-18T15:08:33.103 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:08:33.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=274877906946 2024-09-18T15:08:33.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 274877906946' 2024-09-18T15:08:33.105 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 274877906946 2024-09-18T15:08:33.106 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:08:33.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 274877906946 -lt 274877906946 2024-09-18T15:08:33.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:08:33.446 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-146028888076 2024-09-18T15:08:33.446 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:08:33.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:08:33.449 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-146028888076 2024-09-18T15:08:33.450 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:08:33.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=146028888076 2024-09-18T15:08:33.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 146028888076' 2024-09-18T15:08:33.451 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 146028888076 2024-09-18T15:08:33.452 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:08:33.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 146028888076 -lt 146028888076 2024-09-18T15:08:33.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:08:33.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:08:33.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:08:34.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:08:34.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:08:34.201 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:08:34.201 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:08:34.201 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:08:34.201 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:08:34.202 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:08:34.202 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:08:34.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:08:34.505 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:08:34.506 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:08:34.506 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:08:34.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:08:34.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:08:34.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:08:34.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:301: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-09-18T15:08:34.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-09-18T15:08:34.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T15:08:34.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:08:34.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:08:35.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:301: auto_repair_erasure_coded: local pgid=2.0 2024-09-18T15:08:35.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-09-18T15:08:35.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:08:35.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:08:35.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:08:35.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:08:35.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-09-18T15:08:20.690403+0000 2024-09-18T15:08:35.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=2.0 2024-09-18T15:08:35.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:08:20.690403+0000 2024-09-18T15:08:35.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:08:35.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:08:35.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:08:35.587 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:08:35.587 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:08:35.587 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:08:35.588 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:08:35.588 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:08:35.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:08:20.690403+0000 '>' 2024-09-18T15:08:20.690403+0000 2024-09-18T15:08:35.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:08:36.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:08:36.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:08:36.892 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:08:36.892 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:08:36.892 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:08:36.892 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:08:36.893 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:08:37.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:08:32.583054+0000 '>' 2024-09-18T15:08:20.690403+0000 2024-09-18T15:08:37.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:08:37.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: wait_for_clean 2024-09-18T15:08:37.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:08:37.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:08:37.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:08:37.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:08:37.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:08:37.387 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:08:37.387 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:08:37.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:08:37.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:08:37.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:08:37.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:08:37.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:08:37.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:08:37.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:08:37.599 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:08:37.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:08:37.915 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:08:37.915 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:08:37.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:08:37.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:08:37.916 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:08:38.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836513 2024-09-18T15:08:38.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836513 2024-09-18T15:08:38.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513' 2024-09-18T15:08:38.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:08:38.080 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:08:38.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=274877906948 2024-09-18T15:08:38.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 274877906948 2024-09-18T15:08:38.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513 1-274877906948' 2024-09-18T15:08:38.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:08:38.232 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:08:38.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=146028888078 2024-09-18T15:08:38.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 146028888078 2024-09-18T15:08:38.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513 1-274877906948 2-146028888078' 2024-09-18T15:08:38.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:08:38.376 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836513 2024-09-18T15:08:38.376 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:08:38.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:08:38.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836513 2024-09-18T15:08:38.380 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:08:38.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836513 2024-09-18T15:08:38.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-09-18T15:08:38.382 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836513 2024-09-18T15:08:38.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:08:38.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836513 -lt 21474836513 2024-09-18T15:08:38.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:08:38.742 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-274877906948 2024-09-18T15:08:38.742 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:08:38.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:08:38.745 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-274877906948 2024-09-18T15:08:38.745 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:08:38.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=274877906948 2024-09-18T15:08:38.747 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 274877906948 2024-09-18T15:08:38.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 274877906948' 2024-09-18T15:08:38.748 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:08:39.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 274877906948 -lt 274877906948 2024-09-18T15:08:39.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:08:39.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-146028888078 2024-09-18T15:08:39.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:08:39.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:08:39.127 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-146028888078 2024-09-18T15:08:39.127 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:08:39.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=146028888078 2024-09-18T15:08:39.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 146028888078' 2024-09-18T15:08:39.129 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 146028888078 2024-09-18T15:08:39.129 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:08:39.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 146028888078 -lt 146028888078 2024-09-18T15:08:39.439 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:08:39.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:08:39.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:08:39.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:08:39.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:08:39.862 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:08:39.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:08:39.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:08:39.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:08:39.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:08:39.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:08:40.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:08:40.166 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:08:40.166 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:08:40.166 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:08:40.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:08:40.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:08:40.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:08:40.566 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:306: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-09-18T15:08:40.566 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-09-18T15:08:40.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-09-18T15:08:40.567 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-09-18T15:08:40.567 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-09-18T15:08:40.567 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T15:08:40.567 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:08:40.568 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:08:40.909 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-09-18T15:08:40.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:08:40.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-09-18T15:08:41.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:306: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:08:41.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:08:41.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:08:41.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:08:41.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:08:41.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:08:41.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:08:41.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:08:41.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:08:41.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:08:41.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:08:41.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:08:41.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:08:41.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:08:41.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:08:41.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:08:41.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:08:41.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:08:41.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:08:41.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:08:41.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:08:41.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:08:41.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:08:41.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-18T15:08:43.730 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-18T15:08:43.730 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-09-18T15:08:43.731 INFO:tasks.workunit.client.0.smithi046.stdout:_ 2024-09-18T15:08:43.731 INFO:tasks.workunit.client.0.smithi046.stdout:hinfo_key 2024-09-18T15:08:43.731 INFO:tasks.workunit.client.0.smithi046.stdout:snapset 2024-09-18T15:08:44.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:08:44.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:08:44.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:08:44.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:08:44.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:08:44.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:08:44.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:08:44.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:08:44.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:08:44.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:08:44.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:08:44.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:08:44.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:08:44.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:08:44.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:08:44.269 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:08:44.269 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:08:44.270 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:08:44.270 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:08:44.270 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:08:44.270 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:08:44.270 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:08:44.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:08:44.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:08:44.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:08:44.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:08:44.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:08:44.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:08:44.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:08:44.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:08:44.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:08:44.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:08:44.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:08:44.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:08:44.272 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:08:44.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:08:44.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:08:44.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:08:44.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:08:44.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:08:44.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:08:44.309 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:08:44.303+0000 7f0f47b03540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:08:44.313 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:08:44.307+0000 7f0f47b03540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:08:44.320 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:08:44.307+0000 7f0f47b03540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:08:44.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:08:44.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:08:44.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:08:44.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:08:44.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:08:44.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:08:44.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:08:44.591 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:08:44.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:08:44.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:08:44.847 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:08:44.839+0000 7f0f47b03540 -1 Falling back to public interface 2024-09-18T15:08:44.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:08:45.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:08:45.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:08:45.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:08:45.891 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:08:45.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:08:45.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:08:46.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:08:47.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:08:47.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:08:47.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:08:47.213 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:08:47.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:08:47.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:08:47.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:08:48.121 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:08:48.115+0000 7f0f47b03540 -1 osd.1 73 log_to_monitors true 2024-09-18T15:08:48.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:08:48.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:08:48.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:08:48.585 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:08:48.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:08:48.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:08:48.932 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 81 up_thru 64 down_at 74 last_clean_interval [64,73) [v2:127.0.0.1:6810/215662,v1:127.0.0.1:6811/215662] [v2:127.0.0.1:6812/215662,v1:127.0.0.1:6813/215662] exists,up 786eb5db-6fdf-4b87-87ac-79f18adc37d3 2024-09-18T15:08:48.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:08:48.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:08:48.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:08:48.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:08:48.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:08:48.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:08:48.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:08:48.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:08:48.934 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:08:48.934 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:08:48.935 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:08:48.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:08:48.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:08:48.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:08:49.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:08:49.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:08:49.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:08:49.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:08:49.137 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:08:49.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:08:49.672 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:08:49.672 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:08:49.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:08:49.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:08:49.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:08:49.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836516 2024-09-18T15:08:49.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836516 2024-09-18T15:08:49.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836516' 2024-09-18T15:08:49.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:08:49.895 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:08:50.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=347892350978 2024-09-18T15:08:50.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 347892350978 2024-09-18T15:08:50.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836516 1-347892350978' 2024-09-18T15:08:50.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:08:50.133 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:08:50.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=146028888082 2024-09-18T15:08:50.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 146028888082 2024-09-18T15:08:50.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836516 1-347892350978 2-146028888082' 2024-09-18T15:08:50.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:08:50.269 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836516 2024-09-18T15:08:50.269 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:08:50.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:08:50.272 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836516 2024-09-18T15:08:50.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:08:50.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836516 2024-09-18T15:08:50.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-09-18T15:08:50.275 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836516 2024-09-18T15:08:50.276 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:08:50.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836515 -lt 21474836516 2024-09-18T15:08:50.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:08:51.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:08:51.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:08:51.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836516 -lt 21474836516 2024-09-18T15:08:51.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:08:51.918 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-347892350978 2024-09-18T15:08:51.918 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:08:51.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:08:51.921 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-347892350978 2024-09-18T15:08:51.921 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:08:51.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=347892350978 2024-09-18T15:08:51.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 347892350978' 2024-09-18T15:08:51.923 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 347892350978 2024-09-18T15:08:51.924 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:08:52.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 347892350978 -lt 347892350978 2024-09-18T15:08:52.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:08:52.235 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-146028888082 2024-09-18T15:08:52.236 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:08:52.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:08:52.239 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-146028888082 2024-09-18T15:08:52.239 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:08:52.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=146028888082 2024-09-18T15:08:52.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 146028888082' 2024-09-18T15:08:52.241 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 146028888082 2024-09-18T15:08:52.242 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:08:52.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 146028888082 -lt 146028888082 2024-09-18T15:08:52.576 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:08:52.577 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:08:52.577 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:08:53.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:08:53.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:08:53.034 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:08:53.035 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:08:53.035 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:08:53.035 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:08:53.035 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:08:53.035 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:08:53.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:08:53.487 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:08:53.487 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:08:53.488 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:08:53.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:08:53.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:08:53.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:08:53.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:307: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-18T15:08:53.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:308: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-18T15:08:53.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T15:08:53.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:08:53.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:08:53.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:08:53.917 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:08:53.917 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:08:53.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:08:53.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:08:53.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:08:54.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:08:54.037 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:08:54.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:08:54.038 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:08:54.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:08:54.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:08:54.041 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:08:54.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:08:54.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:08:54.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:08:54.044 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:08:54.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:08:54.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:08:54.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:08:54.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:08:54.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:08:54.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:08:54.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:08:54.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:08:54.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:08:54.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T15:08:54.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T15:08:54.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T15:08:54.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T15:08:54.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:08:54.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:08:54.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:08:54.077 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:08:54.077 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:08:54.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:08:54.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:08:54.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:08:54.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:08:54.083 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:08:54.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:08:54.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:08:54.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:08:54.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:08:54.087 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:08:54.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:08:54.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:08:54.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:08:54.090 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:08:54.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:08:54.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:08:54.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:08:54.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:08:54.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:08:54.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:08:54.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:08:54.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:08:54.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:08:54.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T15:08:54.101 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:08:54.101 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:08:54.101 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:08:54.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T15:08:54.104 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:08:54.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:08:54.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:08:54.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T15:08:54.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_corrupt_and_repair_jerasure_appends td/osd-scrub-repair 2024-09-18T15:08:54.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:820: TEST_corrupt_and_repair_jerasure_appends: corrupt_and_repair_jerasure td/osd-scrub-repair false 2024-09-18T15:08:54.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-09-18T15:08:54.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local allow_overwrites=false 2024-09-18T15:08:54.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:801: corrupt_and_repair_jerasure: local poolname=ecpool 2024-09-18T15:08:54.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-09-18T15:08:54.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T15:08:54.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:08:54.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:08:54.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:08:54.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T15:08:54.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-18T15:08:54.476 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:08:54.476 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:08:54.476 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:08:54.477 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:08:54.477 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:08:54.477 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:08:54.477 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:08:54.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T15:08:54.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:08:54.800 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:08:54.800 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:08:54.800 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:08:54.800 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:08:54.801 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:08:54.801 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:08:54.801 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:08:54.801 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:08:54.802 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:08:54.802 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:08:54.803 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:08:54.803 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:08:54.803 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:08:54.803 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T15:08:54.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:08:54.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:08:54.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:08:54.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:08:54.895 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:08:54.895 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:08:54.895 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:08:54.895 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:08:54.895 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:08:54.896 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:08:54.896 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:08:54.896 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:08:54.897 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:08:54.897 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T15:08:54.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-09-18T15:08:54.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T15:08:54.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:08:54.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:08:54.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:08:54.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T15:08:54.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:08:55.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:08:55.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:08:55.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:08:55.851 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:08:55.851 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:08:55.851 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:08:55.851 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:08:55.853 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:08:55.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:08:55.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: seq 0 3 2024-09-18T15:08:55.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-18T15:08:55.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: corrupt_and_repair_jerasure: '[' false = true ']' 2024-09-18T15:08:55.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:809: corrupt_and_repair_jerasure: run_osd_filestore td/osd-scrub-repair 0 2024-09-18T15:08:55.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T15:08:55.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T15:08:55.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=0 2024-09-18T15:08:55.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T15:08:55.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:08:55.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:08:55.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:08:55.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:08:55.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:08:55.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:08:55.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:08:55.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T15:08:55.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T15:08:55.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:08:55.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T15:08:55.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:08:55.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:08:55.894 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:08:55.894 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:08:55.894 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:08:55.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:08:55.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:08:55.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T15:08:55.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T15:08:55.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T15:08:55.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:08:55.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:08:55.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:08:55.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:08:55.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:08:55.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T15:08:55.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T15:08:55.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:08:55.897 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T15:08:55.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=b020c6d9-ff65-4a55-9719-96e8fc979d7c 2024-09-18T15:08:55.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd2 b020c6d9-ff65-4a55-9719-96e8fc979d7c' 2024-09-18T15:08:55.898 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 b020c6d9-ff65-4a55-9719-96e8fc979d7c 2024-09-18T15:08:55.899 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T15:08:55.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQAH7epmzuF2NhAAIwO5MKxmYi3lthopFoTRpA== 2024-09-18T15:08:55.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQAH7epmzuF2NhAAIwO5MKxmYi3lthopFoTRpA=="}' 2024-09-18T15:08:55.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new b020c6d9-ff65-4a55-9719-96e8fc979d7c -i td/osd-scrub-repair/0/new.json 2024-09-18T15:08:56.192 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:08:56.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/0/new.json 2024-09-18T15:08:56.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAH7epmzuF2NhAAIwO5MKxmYi3lthopFoTRpA== --osd-uuid b020c6d9-ff65-4a55-9719-96e8fc979d7c --osd-objectstore=filestore 2024-09-18T15:08:56.239 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:08:56.231+0000 7f9e1446a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:08:56.285 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:08:56.279+0000 7f9e1446a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:08:56.287 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:08:56.279+0000 7f9e1446a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:08:56.702 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:08:56.695+0000 7f9e1446a540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:08:57.232 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:08:57.223+0000 7f9e1446a540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:08:57.232 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:08:57.223+0000 7f9e1446a540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:08:57.232 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:08:57.223+0000 7f9e1446a540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:08:57.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T15:08:57.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T15:08:57.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd0 key to auth repository 2024-09-18T15:08:57.808 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T15:08:57.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:08:57.981 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: key for osd.0 not found in provided keyring 2024-09-18T15:08:57.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.0 2024-09-18T15:08:57.984 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T15:08:57.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:08:57.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T15:08:57.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T15:08:57.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T15:08:58.006 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:08:57.999+0000 7f466a858540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:08:58.008 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:08:58.003+0000 7f466a858540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:08:58.012 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:08:58.003+0000 7f466a858540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:08:58.012 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:08:58.003+0000 7f466a858540 -1 Falling back to public interface 2024-09-18T15:08:58.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 0 2024-09-18T15:08:58.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:08:58.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:08:58.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:08:58.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:08:58.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:08:58.352 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:08:58.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:08:58.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:08:58.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:08:58.557 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:08:58.551+0000 7f466a858540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:08:58.557 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:08:58.551+0000 7f466a858540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:08:58.557 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:08:58.551+0000 7f466a858540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:08:58.566 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:08:58.559+0000 7f466a858540 -1 osd.0 0 log_to_monitors true 2024-09-18T15:08:58.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:08:59.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:08:59.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:08:59.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:08:59.620 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:08:59.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:08:59.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:08:59.817 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/216964,v1:127.0.0.1:6801/216964] [v2:127.0.0.1:6802/216964,v1:127.0.0.1:6803/216964] exists,up b020c6d9-ff65-4a55-9719-96e8fc979d7c 2024-09-18T15:08:59.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:08:59.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:08:59.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:08:59.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-18T15:08:59.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: corrupt_and_repair_jerasure: '[' false = true ']' 2024-09-18T15:08:59.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:809: corrupt_and_repair_jerasure: run_osd_filestore td/osd-scrub-repair 1 2024-09-18T15:08:59.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T15:08:59.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T15:08:59.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=1 2024-09-18T15:08:59.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T15:08:59.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:08:59.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:08:59.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:08:59.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:08:59.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:08:59.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:08:59.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:08:59.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T15:08:59.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T15:08:59.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:08:59.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T15:08:59.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:08:59.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:08:59.822 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:08:59.822 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:08:59.822 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:08:59.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:08:59.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:08:59.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T15:08:59.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T15:08:59.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T15:08:59.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:08:59.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:08:59.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:08:59.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:08:59.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:08:59.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T15:08:59.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T15:08:59.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:08:59.824 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T15:08:59.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=115f2623-03c8-4519-b5cd-6dfb4555432f 2024-09-18T15:08:59.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd2 115f2623-03c8-4519-b5cd-6dfb4555432f' 2024-09-18T15:08:59.825 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 115f2623-03c8-4519-b5cd-6dfb4555432f 2024-09-18T15:08:59.825 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T15:08:59.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQAL7epmYo3cMRAAyYB3h4CX+nux1BWBV2cOlA== 2024-09-18T15:08:59.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQAL7epmYo3cMRAAyYB3h4CX+nux1BWBV2cOlA=="}' 2024-09-18T15:08:59.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 115f2623-03c8-4519-b5cd-6dfb4555432f -i td/osd-scrub-repair/1/new.json 2024-09-18T15:09:00.139 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:09:00.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/1/new.json 2024-09-18T15:09:00.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAL7epmYo3cMRAAyYB3h4CX+nux1BWBV2cOlA== --osd-uuid 115f2623-03c8-4519-b5cd-6dfb4555432f --osd-objectstore=filestore 2024-09-18T15:09:00.189 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:00.179+0000 7f0f73306540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:00.196 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:00.187+0000 7f0f73306540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:00.198 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:00.191+0000 7f0f73306540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:00.724 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:00.715+0000 7f0f73306540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:09:01.337 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:01.331+0000 7f0f73306540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:09:01.345 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:01.339+0000 7f0f73306540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:09:01.345 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:01.339+0000 7f0f73306540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:09:02.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-18T15:09:02.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T15:09:02.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd1 key to auth repository 2024-09-18T15:09:02.475 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T15:09:02.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:09:02.851 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: key for osd.1 not found in provided keyring 2024-09-18T15:09:02.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.1 2024-09-18T15:09:02.854 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T15:09:02.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T15:09:02.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:09:02.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T15:09:02.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T15:09:02.884 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:02.875+0000 7fb12e436540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:02.886 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:02.879+0000 7fb12e436540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:02.892 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:02.879+0000 7fb12e436540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:02.893 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:02.883+0000 7fb12e436540 -1 Falling back to public interface 2024-09-18T15:09:03.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 1 2024-09-18T15:09:03.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:09:03.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:09:03.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:09:03.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:09:03.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:03.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:09:03.247 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:09:03.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:03.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:09:03.487 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:03.479+0000 7fb12e436540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:09:03.492 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:03.483+0000 7fb12e436540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:09:03.492 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:03.483+0000 7fb12e436540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:09:03.498 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:03.491+0000 7fb12e436540 -1 osd.1 0 log_to_monitors true 2024-09-18T15:09:03.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:09:04.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:09:04.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:04.581 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:09:04.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:09:04.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:04.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:09:04.892 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/217346,v1:127.0.0.1:6811/217346] [v2:127.0.0.1:6812/217346,v1:127.0.0.1:6813/217346] exists,up 115f2623-03c8-4519-b5cd-6dfb4555432f 2024-09-18T15:09:04.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:09:04.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:09:04.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:09:04.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-18T15:09:04.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: corrupt_and_repair_jerasure: '[' false = true ']' 2024-09-18T15:09:04.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:809: corrupt_and_repair_jerasure: run_osd_filestore td/osd-scrub-repair 2 2024-09-18T15:09:04.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T15:09:04.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T15:09:04.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=2 2024-09-18T15:09:04.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T15:09:04.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:09:04.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:09:04.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:09:04.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:09:04.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:09:04.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T15:09:04.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T15:09:04.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T15:09:04.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T15:09:04.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:09:04.896 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T15:09:04.896 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:09:04.896 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:09:04.897 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:09:04.898 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:09:04.898 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:09:04.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:09:04.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:09:04.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T15:09:04.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T15:09:04.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T15:09:04.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:09:04.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:09:04.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:09:04.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:09:04.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:09:04.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T15:09:04.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T15:09:04.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/2 2024-09-18T15:09:04.899 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T15:09:04.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=8880f823-a5a9-4299-a2ed-dbe0431c26c7 2024-09-18T15:09:04.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd2 8880f823-a5a9-4299-a2ed-dbe0431c26c7' 2024-09-18T15:09:04.900 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 8880f823-a5a9-4299-a2ed-dbe0431c26c7 2024-09-18T15:09:04.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T15:09:04.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQAQ7epmzNRzNhAAPHJltXwfggD2jv+XW5l33Q== 2024-09-18T15:09:04.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQAQ7epmzNRzNhAAPHJltXwfggD2jv+XW5l33Q=="}' 2024-09-18T15:09:04.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 8880f823-a5a9-4299-a2ed-dbe0431c26c7 -i td/osd-scrub-repair/2/new.json 2024-09-18T15:09:05.306 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:09:05.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/2/new.json 2024-09-18T15:09:05.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAQ7epmzNRzNhAAPHJltXwfggD2jv+XW5l33Q== --osd-uuid 8880f823-a5a9-4299-a2ed-dbe0431c26c7 --osd-objectstore=filestore 2024-09-18T15:09:05.361 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:05.355+0000 7ff34416e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:05.364 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:05.355+0000 7ff34416e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:05.366 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:05.359+0000 7ff34416e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:06.074 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:06.067+0000 7ff34416e540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:09:06.830 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:06.823+0000 7ff34416e540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:09:06.830 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:06.823+0000 7ff34416e540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:09:06.831 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:06.823+0000 7ff34416e540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:09:07.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-18T15:09:07.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T15:09:07.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd2 key to auth repository 2024-09-18T15:09:07.593 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T15:09:07.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:09:07.988 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T15:09:07.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.2 2024-09-18T15:09:07.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:09:07.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T15:09:07.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T15:09:07.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T15:09:08.015 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:08.007+0000 7f63c5f8c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:08.037 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:08.031+0000 7f63c5f8c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:08.039 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:08.031+0000 7f63c5f8c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:08.041 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:08.035+0000 7f63c5f8c540 -1 Falling back to public interface 2024-09-18T15:09:08.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 2 2024-09-18T15:09:08.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:09:08.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:09:08.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:09:08.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:09:08.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:08.295 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:09:08.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:09:08.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:08.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:09:08.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:09:08.739 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:08.731+0000 7f63c5f8c540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:09:08.754 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:08.747+0000 7f63c5f8c540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:09:08.754 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:08.747+0000 7f63c5f8c540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:09:08.763 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:08.755+0000 7f63c5f8c540 -1 osd.2 0 log_to_monitors true 2024-09-18T15:09:09.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:09:09.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:09.651 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:09:09.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:09:09.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:09.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:09:10.079 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/217630,v1:127.0.0.1:6819/217630] [v2:127.0.0.1:6820/217630,v1:127.0.0.1:6821/217630] exists,up 8880f823-a5a9-4299-a2ed-dbe0431c26c7 2024-09-18T15:09:10.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:09:10.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:09:10.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:09:10.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-18T15:09:10.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: corrupt_and_repair_jerasure: '[' false = true ']' 2024-09-18T15:09:10.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:809: corrupt_and_repair_jerasure: run_osd_filestore td/osd-scrub-repair 3 2024-09-18T15:09:10.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T15:09:10.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T15:09:10.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=3 2024-09-18T15:09:10.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T15:09:10.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:09:10.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:09:10.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:09:10.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:09:10.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:09:10.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-18T15:09:10.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-18T15:09:10.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T15:09:10.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T15:09:10.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:09:10.083 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T15:09:10.083 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:09:10.083 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:09:10.085 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:09:10.085 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:09:10.085 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:09:10.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:09:10.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:09:10.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T15:09:10.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T15:09:10.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T15:09:10.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:09:10.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:09:10.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:09:10.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:09:10.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:09:10.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T15:09:10.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T15:09:10.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/3 2024-09-18T15:09:10.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T15:09:10.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=d82b14ff-15f5-422f-9dfe-8c6bfd9425b4 2024-09-18T15:09:10.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd2 d82b14ff-15f5-422f-9dfe-8c6bfd9425b4' 2024-09-18T15:09:10.087 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 d82b14ff-15f5-422f-9dfe-8c6bfd9425b4 2024-09-18T15:09:10.087 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T15:09:10.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQAW7epmx2UdBhAAZonH8pCe8Fkc7jK6viERfw== 2024-09-18T15:09:10.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQAW7epmx2UdBhAAZonH8pCe8Fkc7jK6viERfw=="}' 2024-09-18T15:09:10.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new d82b14ff-15f5-422f-9dfe-8c6bfd9425b4 -i td/osd-scrub-repair/3/new.json 2024-09-18T15:09:10.498 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:09:10.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/3/new.json 2024-09-18T15:09:10.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAW7epmx2UdBhAAZonH8pCe8Fkc7jK6viERfw== --osd-uuid d82b14ff-15f5-422f-9dfe-8c6bfd9425b4 --osd-objectstore=filestore 2024-09-18T15:09:10.546 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:10.539+0000 7f29b8e2e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:10.550 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:10.543+0000 7f29b8e2e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:10.552 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:10.543+0000 7f29b8e2e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:11.305 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:11.299+0000 7f29b8e2e540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:09:12.013 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:12.007+0000 7f29b8e2e540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:09:12.029 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:12.023+0000 7f29b8e2e540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:09:12.029 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:12.023+0000 7f29b8e2e540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:09:12.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-18T15:09:12.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T15:09:12.743 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd3 key to auth repository 2024-09-18T15:09:12.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd3 key to auth repository 2024-09-18T15:09:12.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:09:13.129 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: key for osd.3 not found in provided keyring 2024-09-18T15:09:13.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.3 2024-09-18T15:09:13.132 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.3 2024-09-18T15:09:13.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T15:09:13.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:09:13.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T15:09:13.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T15:09:13.161 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:13.155+0000 7f65fb221540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:13.165 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:13.159+0000 7f65fb221540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:13.168 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:13.159+0000 7f65fb221540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:13.169 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:13.163+0000 7f65fb221540 -1 Falling back to public interface 2024-09-18T15:09:13.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 3 2024-09-18T15:09:13.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:09:13.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:09:13.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:09:13.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:09:13.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:13.531 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:09:13.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:09:13.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:13.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:09:13.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:09:13.857 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:13.851+0000 7f65fb221540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:09:13.920 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:13.911+0000 7f65fb221540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:09:13.921 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:13.911+0000 7f65fb221540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:09:13.929 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:13.923+0000 7f65fb221540 -1 osd.3 0 log_to_monitors true 2024-09-18T15:09:14.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:09:14.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:14.857 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:09:14.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:09:14.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:14.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:09:15.205 INFO:tasks.workunit.client.0.smithi046.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/217913,v1:127.0.0.1:6827/217913] [v2:127.0.0.1:6828/217913,v1:127.0.0.1:6829/217913] exists,up d82b14ff-15f5-422f-9dfe-8c6bfd9425b4 2024-09-18T15:09:15.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:09:15.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:09:15.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:09:15.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:812: corrupt_and_repair_jerasure: create_rbd_pool 2024-09-18T15:09:15.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T15:09:15.513 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' does not exist 2024-09-18T15:09:15.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T15:09:15.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T15:09:16.014 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' created 2024-09-18T15:09:16.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:09:17.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T15:09:19.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:813: corrupt_and_repair_jerasure: wait_for_clean 2024-09-18T15:09:19.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:09:19.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:09:19.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:09:19.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:09:19.868 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:09:19.869 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:09:19.869 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:09:19.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:09:19.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:09:19.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:09:20.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:09:20.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:09:20.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:09:20.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:09:20.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:09:20.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:09:20.411 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:09:20.411 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:09:20.411 INFO:tasks.workunit.client.0.smithi046.stderr:3' 2024-09-18T15:09:20.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:09:20.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:09:20.412 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:09:20.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836485 2024-09-18T15:09:20.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836485 2024-09-18T15:09:20.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836485' 2024-09-18T15:09:20.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:09:20.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:09:20.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672965 2024-09-18T15:09:20.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672965 2024-09-18T15:09:20.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836485 1-42949672965' 2024-09-18T15:09:20.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:09:20.737 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:09:20.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509444 2024-09-18T15:09:20.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509444 2024-09-18T15:09:20.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836485 1-42949672965 2-64424509444' 2024-09-18T15:09:20.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:09:20.881 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:09:21.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345923 2024-09-18T15:09:21.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345923 2024-09-18T15:09:21.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836485 1-42949672965 2-64424509444 3-85899345923' 2024-09-18T15:09:21.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:09:21.020 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836485 2024-09-18T15:09:21.021 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:09:21.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:09:21.023 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836485 2024-09-18T15:09:21.023 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:09:21.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836485 2024-09-18T15:09:21.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-09-18T15:09:21.026 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836485 2024-09-18T15:09:21.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:09:21.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836485 -lt 21474836485 2024-09-18T15:09:21.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:09:21.348 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672965 2024-09-18T15:09:21.348 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:09:21.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:09:21.351 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672965 2024-09-18T15:09:21.352 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:09:21.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672965 2024-09-18T15:09:21.354 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672965 2024-09-18T15:09:21.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-18T15:09:21.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:09:21.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672965 -lt 42949672965 2024-09-18T15:09:21.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:09:21.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509444 2024-09-18T15:09:21.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:09:21.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:09:21.661 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509444 2024-09-18T15:09:21.662 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:09:21.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509444 2024-09-18T15:09:21.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-09-18T15:09:21.664 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509444 2024-09-18T15:09:21.665 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:09:22.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509444 -lt 64424509444 2024-09-18T15:09:22.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:09:22.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345923 2024-09-18T15:09:22.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:09:22.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:09:22.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345923 2024-09-18T15:09:22.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:09:22.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345923 2024-09-18T15:09:22.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345923' 2024-09-18T15:09:22.063 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.3 seq 85899345923 2024-09-18T15:09:22.063 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:09:22.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345923 -lt 85899345923 2024-09-18T15:09:22.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:09:22.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:09:22.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:09:22.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T15:09:22.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:09:22.785 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:09:22.785 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:09:22.785 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:09:22.785 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:09:22.785 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:09:22.786 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:09:23.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T15:09:23.120 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:09:23.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:09:23.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:09:23.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T15:09:23.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:09:23.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:09:23.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:815: corrupt_and_repair_jerasure: create_ec_pool ecpool false k=2 m=2 2024-09-18T15:09:23.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: create_ec_pool: local pool_name=ecpool 2024-09-18T15:09:23.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: create_ec_pool: shift 2024-09-18T15:09:23.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: create_ec_pool: local allow_overwrites=false 2024-09-18T15:09:23.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: create_ec_pool: shift 2024-09-18T15:09:23.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-09-18T15:09:23.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-18T15:09:23.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-18T15:09:24.989 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'ecpool' created 2024-09-18T15:09:25.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:09:26.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: create_ec_pool: '[' false = true ']' 2024-09-18T15:09:26.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: create_ec_pool: wait_for_clean 2024-09-18T15:09:26.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:09:26.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:09:26.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:09:26.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:09:26.005 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:09:26.006 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:09:26.006 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:09:26.007 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:09:26.007 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:09:26.007 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:09:26.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:09:26.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:09:26.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:09:26.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:09:26.226 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:09:26.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:09:26.564 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:09:26.564 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:09:26.564 INFO:tasks.workunit.client.0.smithi046.stderr:3' 2024-09-18T15:09:26.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:09:26.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:09:26.564 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:09:26.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836488 2024-09-18T15:09:26.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836488 2024-09-18T15:09:26.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836488' 2024-09-18T15:09:26.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:09:26.703 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:09:26.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672967 2024-09-18T15:09:26.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672967 2024-09-18T15:09:26.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836488 1-42949672967' 2024-09-18T15:09:26.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:09:26.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:09:26.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509446 2024-09-18T15:09:26.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509446 2024-09-18T15:09:26.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836488 1-42949672967 2-64424509446' 2024-09-18T15:09:26.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:09:26.992 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:09:27.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345925 2024-09-18T15:09:27.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345925 2024-09-18T15:09:27.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836488 1-42949672967 2-64424509446 3-85899345925' 2024-09-18T15:09:27.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:09:27.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836488 2024-09-18T15:09:27.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:09:27.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:09:27.174 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836488 2024-09-18T15:09:27.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:09:27.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836488 2024-09-18T15:09:27.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-09-18T15:09:27.177 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836488 2024-09-18T15:09:27.178 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:09:27.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836488 -lt 21474836488 2024-09-18T15:09:27.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:09:27.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672967 2024-09-18T15:09:27.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:09:27.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:09:27.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672967 2024-09-18T15:09:27.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:09:27.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672967 2024-09-18T15:09:27.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-09-18T15:09:27.496 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672967 2024-09-18T15:09:27.496 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:09:27.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672967 -lt 42949672967 2024-09-18T15:09:27.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:09:27.862 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509446 2024-09-18T15:09:27.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:09:27.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:09:27.866 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509446 2024-09-18T15:09:27.866 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:09:27.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509446 2024-09-18T15:09:27.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-09-18T15:09:27.869 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509446 2024-09-18T15:09:27.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:09:28.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509446 -lt 64424509446 2024-09-18T15:09:28.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:09:28.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345925 2024-09-18T15:09:28.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:09:28.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:09:28.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345925 2024-09-18T15:09:28.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:09:28.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345925 2024-09-18T15:09:28.219 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.3 seq 85899345925 2024-09-18T15:09:28.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345925' 2024-09-18T15:09:28.219 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:09:28.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345925 -lt 85899345925 2024-09-18T15:09:28.558 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:09:28.559 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:09:28.559 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:09:28.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:09:28.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:09:28.980 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:09:28.980 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:09:28.980 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:09:28.980 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:09:28.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:09:28.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:09:29.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:09:29.328 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:09:29.329 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:09:29.329 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:09:29.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:09:29.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:09:29.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:09:29.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: create_ec_pool: return 0 2024-09-18T15:09:29.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:816: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-09-18T15:09:29.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-18T15:09:29.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-09-18T15:09:29.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:250: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-18T15:09:29.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:09:29.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2024-09-18T15:09:29.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=SOMETHING 2024-09-18T15:09:29.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:09:29.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:09:29.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:09:30.537 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:09:30.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:09:31.108 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:09:31.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:09:31.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:09:31.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-18T15:09:31.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:252: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-09-18T15:09:31.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-09-18T15:09:31.282 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T15:09:31.282 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:09:31.282 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:09:31.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:252: corrupt_and_repair_erasure_coded: local primary=3 2024-09-18T15:09:31.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: osds=($(get_osds $poolname SOMETHING | sed -e "s/$primary//")) 2024-09-18T15:09:31.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-09-18T15:09:31.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-09-18T15:09:31.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-18T15:09:31.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-09-18T15:09:31.621 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:09:31.622 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:09:31.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:09:31.997 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:09:31.997 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:09:31.997 INFO:tasks.workunit.client.0.smithi046.stderr:0' 2024-09-18T15:09:31.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 0 2024-09-18T15:09:32.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local -a osds 2024-09-18T15:09:32.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-09-18T15:09:32.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-09-18T15:09:32.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-09-18T15:09:32.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:225: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-18T15:09:32.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local poolname=ecpool 2024-09-18T15:09:32.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local osd=3 2024-09-18T15:09:32.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:09:32.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:09:32.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:09:32.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-18T15:09:32.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:09:32.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:09:32.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:09:32.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:09:32.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-18T15:09:32.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:09:32.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-18T15:09:32.001 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:09:32.001 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:09:32.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:09:32.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:09:32.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:09:32.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:09:32.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:09:32.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:09:32.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:09:32.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-18T15:09:32.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:09:32.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:09:32.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-18T15:09:33.217 INFO:tasks.workunit.client.0.smithi046.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-09-18T15:09:33.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-18T15:09:33.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:09:33.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:09:33.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T15:09:33.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:09:33.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:09:33.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:09:33.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:09:33.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:09:33.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:09:33.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-18T15:09:33.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-18T15:09:33.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:09:33.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:09:33.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:09:33.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:09:33.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:09:33.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:09:33.655 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:09:33.656 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:09:33.656 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:09:33.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:09:33.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:09:33.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:09:33.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:09:33.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:09:33.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:09:33.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:09:33.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:09:33.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:09:33.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:09:33.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:09:33.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-18T15:09:33.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T15:09:33.659 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.3 2024-09-18T15:09:33.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:09:33.660 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-18T15:09:33.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T15:09:33.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:09:33.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:09:33.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:09:33.690 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:33.683+0000 7f7948638540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:33.693 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:33.687+0000 7f7948638540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:33.695 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:33.687+0000 7f7948638540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:33.697 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:33.691+0000 7f7948638540 -1 Falling back to public interface 2024-09-18T15:09:34.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T15:09:34.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:09:34.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:09:34.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:09:34.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:09:34.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:34.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:09:34.015 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:09:34.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:34.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:09:34.193 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:34.187+0000 7f7948638540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:09:34.193 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:34.187+0000 7f7948638540 -1 journal do_read_entry(290816): bad header magic 2024-09-18T15:09:34.194 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:34.187+0000 7f7948638540 -1 journal do_read_entry(290816): bad header magic 2024-09-18T15:09:34.205 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:34.199+0000 7f7948638540 -1 osd.3 36 log_to_monitors true 2024-09-18T15:09:34.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:09:35.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:09:35.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:35.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:09:35.392 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:09:35.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:35.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:09:35.707 INFO:tasks.workunit.client.0.smithi046.stderr:osd.3 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [20,36) [v2:127.0.0.1:6826/219189,v1:127.0.0.1:6827/219189] [v2:127.0.0.1:6828/219189,v1:127.0.0.1:6829/219189] exists,up d82b14ff-15f5-422f-9dfe-8c6bfd9425b4 2024-09-18T15:09:35.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:09:35.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:09:35.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:09:35.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:09:35.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:09:35.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:09:35.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:09:35.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:09:35.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:09:35.708 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:09:35.709 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:09:35.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:09:35.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:09:35.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:09:35.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:09:35.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:09:35.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:09:35.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:09:35.936 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:09:36.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:09:36.245 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:09:36.245 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:09:36.245 INFO:tasks.workunit.client.0.smithi046.stderr:3' 2024-09-18T15:09:36.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:09:36.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:09:36.246 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:09:36.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836490 2024-09-18T15:09:36.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836490 2024-09-18T15:09:36.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836490' 2024-09-18T15:09:36.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:09:36.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:09:36.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672970 2024-09-18T15:09:36.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672970 2024-09-18T15:09:36.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836490 1-42949672970' 2024-09-18T15:09:36.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:09:36.555 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:09:36.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509449 2024-09-18T15:09:36.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509449 2024-09-18T15:09:36.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836490 1-42949672970 2-64424509449' 2024-09-18T15:09:36.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:09:36.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:09:36.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691842 2024-09-18T15:09:36.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691842 2024-09-18T15:09:36.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836490 1-42949672970 2-64424509449 3-171798691842' 2024-09-18T15:09:36.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:09:36.829 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836490 2024-09-18T15:09:36.830 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:09:36.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:09:36.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836490 2024-09-18T15:09:36.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:09:36.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836490 2024-09-18T15:09:36.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-09-18T15:09:36.835 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836490 2024-09-18T15:09:36.836 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:09:37.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836489 -lt 21474836490 2024-09-18T15:09:37.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:09:38.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:09:38.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:09:38.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836491 -lt 21474836490 2024-09-18T15:09:38.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:09:38.471 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672970 2024-09-18T15:09:38.472 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:09:38.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:09:38.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672970 2024-09-18T15:09:38.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:09:38.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672970 2024-09-18T15:09:38.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-09-18T15:09:38.476 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 42949672970 2024-09-18T15:09:38.477 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:09:38.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672970 -lt 42949672970 2024-09-18T15:09:38.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:09:38.814 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509449 2024-09-18T15:09:38.814 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:09:38.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:09:38.818 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509449 2024-09-18T15:09:38.818 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:09:38.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509449 2024-09-18T15:09:38.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-18T15:09:38.820 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509449 2024-09-18T15:09:38.820 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:09:39.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509449 -lt 64424509449 2024-09-18T15:09:39.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:09:39.124 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-171798691842 2024-09-18T15:09:39.124 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:09:39.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:09:39.127 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-171798691842 2024-09-18T15:09:39.127 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:09:39.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691842 2024-09-18T15:09:39.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 171798691842' 2024-09-18T15:09:39.129 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.3 seq 171798691842 2024-09-18T15:09:39.130 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:09:39.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691842 -lt 171798691842 2024-09-18T15:09:39.657 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:09:39.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:09:39.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:09:40.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:09:40.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:09:40.078 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:09:40.078 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:09:40.079 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:09:40.079 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:09:40.079 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:09:40.079 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:09:40.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:09:40.403 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:09:40.403 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:09:40.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:09:40.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:09:40.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:09:40.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:09:40.756 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-18T15:09:40.756 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-09-18T15:09:40.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T15:09:40.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:09:40.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:09:41.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: local pg=2.0 2024-09-18T15:09:41.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: repair 2.0 2024-09-18T15:09:41.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-09-18T15:09:41.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-09-18T15:09:41.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:09:41.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:09:41.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:09:41.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:09:41.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T15:09:24.829418+0000 2024-09-18T15:09:41.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-09-18T15:09:41.645 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-18T15:09:41.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-09-18T15:09:24.829418+0000 2024-09-18T15:09:41.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=2.0 2024-09-18T15:09:41.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:09:24.829418+0000 2024-09-18T15:09:41.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:09:41.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:09:41.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:09:41.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:09:41.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:09:41.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:09:41.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:09:41.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:09:41.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:09:24.829418+0000 '>' 2024-09-18T15:09:24.829418+0000 2024-09-18T15:09:41.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:09:42.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:09:42.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:09:42.959 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:09:42.959 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:09:42.959 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:09:42.960 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:09:42.960 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:09:43.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:09:24.829418+0000 '>' 2024-09-18T15:09:24.829418+0000 2024-09-18T15:09:43.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:09:44.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:09:44.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:09:44.305 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:09:44.305 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:09:44.306 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:09:44.306 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:09:44.306 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:09:44.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:09:24.829418+0000 '>' 2024-09-18T15:09:24.829418+0000 2024-09-18T15:09:44.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:09:45.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:09:45.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:09:45.616 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:09:45.617 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:09:45.617 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:09:45.617 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:09:45.617 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:09:45.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:09:24.829418+0000 '>' 2024-09-18T15:09:24.829418+0000 2024-09-18T15:09:45.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:09:46.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:09:46.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:09:46.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:09:46.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:09:46.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:09:46.936 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:09:46.936 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:09:47.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:09:24.829418+0000 '>' 2024-09-18T15:09:24.829418+0000 2024-09-18T15:09:47.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:09:48.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:09:48.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:09:48.293 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:09:48.293 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:09:48.293 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:09:48.294 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:09:48.294 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:09:48.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:09:24.829418+0000 '>' 2024-09-18T15:09:24.829418+0000 2024-09-18T15:09:48.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:09:49.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:09:49.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:09:49.611 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:09:49.611 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:09:49.611 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:09:49.612 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:09:49.612 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:09:49.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:09:24.829418+0000 '>' 2024-09-18T15:09:24.829418+0000 2024-09-18T15:09:49.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:09:50.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:09:50.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:09:50.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:09:50.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:09:50.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:09:50.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:09:50.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:09:51.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:09:45.261277+0000 '>' 2024-09-18T15:09:24.829418+0000 2024-09-18T15:09:51.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:09:51.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:241: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:09:51.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:09:51.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:09:51.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-18T15:09:51.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:09:51.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:09:51.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:09:51.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:09:51.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-18T15:09:51.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:09:51.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-18T15:09:51.252 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:09:51.252 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:09:51.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:09:51.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:09:51.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:09:51.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:09:51.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:09:51.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:09:51.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:09:51.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-18T15:09:51.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:09:51.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:09:51.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-18T15:09:52.483 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-18T15:09:52.483 INFO:tasks.workunit.client.0.smithi046.stdout:_ 2024-09-18T15:09:52.483 INFO:tasks.workunit.client.0.smithi046.stdout:hinfo_key 2024-09-18T15:09:52.483 INFO:tasks.workunit.client.0.smithi046.stdout:snapset 2024-09-18T15:09:52.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-18T15:09:52.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:09:52.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:09:52.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T15:09:52.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:09:52.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:09:52.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:09:52.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:09:52.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:09:52.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:09:52.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-18T15:09:52.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-18T15:09:52.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:09:52.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:09:52.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:09:52.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:09:52.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:09:52.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:09:52.625 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:09:52.625 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:09:52.625 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:09:52.625 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:09:52.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:09:52.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:09:52.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:09:52.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:09:52.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:09:52.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:09:52.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:09:52.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:09:52.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:09:52.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:09:52.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-18T15:09:52.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T15:09:52.626 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.3 2024-09-18T15:09:52.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:09:52.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-18T15:09:52.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T15:09:52.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:09:52.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:09:52.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:09:52.657 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:52.646+0000 7fc52831a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:52.678 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:52.670+0000 7fc52831a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:52.680 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:52.670+0000 7fc52831a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:52.683 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:52.674+0000 7fc52831a540 -1 Falling back to public interface 2024-09-18T15:09:52.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T15:09:52.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:09:52.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:09:52.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:09:52.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:09:52.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:52.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:09:52.941 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:09:52.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:52.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:09:53.044 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:53.034+0000 7fc52831a540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:09:53.051 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:53.042+0000 7fc52831a540 -1 journal do_read_entry(425984): bad header magic 2024-09-18T15:09:53.052 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:53.042+0000 7fc52831a540 -1 journal do_read_entry(425984): bad header magic 2024-09-18T15:09:53.061 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:09:53.054+0000 7fc52831a540 -1 osd.3 42 log_to_monitors true 2024-09-18T15:09:53.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:09:54.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:09:54.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:54.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:09:54.247 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:09:54.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:54.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:09:54.561 INFO:tasks.workunit.client.0.smithi046.stderr:osd.3 up in weight 1 up_from 46 up_thru 46 down_at 43 last_clean_interval [40,42) [v2:127.0.0.1:6826/220127,v1:127.0.0.1:6827/220127] [v2:127.0.0.1:6828/220127,v1:127.0.0.1:6829/220127] exists,up d82b14ff-15f5-422f-9dfe-8c6bfd9425b4 2024-09-18T15:09:54.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:09:54.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:09:54.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:09:54.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:09:54.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:09:54.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:09:54.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:09:54.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:09:54.562 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:09:54.563 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:09:54.563 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:09:54.564 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:09:54.564 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:09:54.564 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:09:54.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:09:54.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:09:54.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:09:54.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:09:54.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:09:55.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:09:55.095 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:09:55.096 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:09:55.096 INFO:tasks.workunit.client.0.smithi046.stderr:3' 2024-09-18T15:09:55.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:09:55.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:09:55.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:09:55.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836495 2024-09-18T15:09:55.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836495 2024-09-18T15:09:55.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836495' 2024-09-18T15:09:55.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:09:55.231 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:09:55.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672975 2024-09-18T15:09:55.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672975 2024-09-18T15:09:55.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836495 1-42949672975' 2024-09-18T15:09:55.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:09:55.367 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:09:55.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509454 2024-09-18T15:09:55.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509454 2024-09-18T15:09:55.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836495 1-42949672975 2-64424509454' 2024-09-18T15:09:55.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:09:55.501 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:09:55.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=197568495618 2024-09-18T15:09:55.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 197568495618 2024-09-18T15:09:55.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836495 1-42949672975 2-64424509454 3-197568495618' 2024-09-18T15:09:55.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:09:55.642 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836495 2024-09-18T15:09:55.642 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:09:55.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:09:55.645 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836495 2024-09-18T15:09:55.645 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:09:55.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836495 2024-09-18T15:09:55.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-09-18T15:09:55.647 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836495 2024-09-18T15:09:55.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:09:55.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836494 -lt 21474836495 2024-09-18T15:09:55.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:09:56.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:09:56.954 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:09:57.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836494 -lt 21474836495 2024-09-18T15:09:57.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:09:58.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-18T15:09:58.274 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:09:58.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836496 -lt 21474836495 2024-09-18T15:09:58.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:09:58.600 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672975 2024-09-18T15:09:58.600 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:09:58.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:09:58.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672975 2024-09-18T15:09:58.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:09:58.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672975 2024-09-18T15:09:58.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-18T15:09:58.605 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 42949672975 2024-09-18T15:09:58.605 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:09:58.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672975 -lt 42949672975 2024-09-18T15:09:58.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:09:58.956 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509454 2024-09-18T15:09:58.956 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:09:58.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:09:58.959 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509454 2024-09-18T15:09:58.959 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:09:58.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509454 2024-09-18T15:09:58.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-09-18T15:09:58.961 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509454 2024-09-18T15:09:58.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:09:59.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509454 -lt 64424509454 2024-09-18T15:09:59.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:09:59.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-197568495618 2024-09-18T15:09:59.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:09:59.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:09:59.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:09:59.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-197568495618 2024-09-18T15:09:59.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=197568495618 2024-09-18T15:09:59.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 197568495618' 2024-09-18T15:09:59.313 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.3 seq 197568495618 2024-09-18T15:09:59.313 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:09:59.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 197568495619 -lt 197568495618 2024-09-18T15:09:59.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:09:59.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:09:59.659 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:10:00.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:10:00.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:10:00.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:10:00.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:10:00.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:10:00.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:10:00.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:10:00.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:10:00.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:10:00.417 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:10:00.420 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:10:00.420 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:10:00.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:10:00.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:10:00.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:10:00.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-18T15:10:00.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-18T15:10:00.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:260: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-09-18T15:10:00.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:225: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-18T15:10:00.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local poolname=ecpool 2024-09-18T15:10:00.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local osd=1 2024-09-18T15:10:00.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:10:00.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:10:00.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:10:00.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:10:00.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:10:00.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:10:00.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:10:00.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:10:00.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:10:00.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:10:00.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:10:00.832 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:10:00.832 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:10:00.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:10:00.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:10:00.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:10:01.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:10:01.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:10:01.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:10:01.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:10:01.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:10:01.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:10:01.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:10:01.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-18T15:10:02.250 INFO:tasks.workunit.client.0.smithi046.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-18T15:10:02.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:10:02.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:10:02.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:10:02.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:10:02.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:10:02.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:10:02.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:10:02.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:10:02.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:10:02.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:10:02.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:10:02.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:10:02.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:10:02.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:10:02.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:10:02.800 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:10:02.800 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:10:02.800 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:10:02.802 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:10:02.802 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:10:02.802 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:10:02.802 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:10:02.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:10:02.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:10:02.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:10:02.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:10:02.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:10:02.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:10:02.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:10:02.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:10:02.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:10:02.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:10:02.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:10:02.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:10:02.804 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:10:02.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:10:02.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:10:02.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:10:02.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:10:02.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:10:02.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:10:02.837 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:10:02.830+0000 7f1e7ded0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:10:02.840 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:10:02.830+0000 7f1e7ded0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:10:02.841 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:10:02.834+0000 7f1e7ded0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:10:02.843 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:10:02.834+0000 7f1e7ded0540 -1 Falling back to public interface 2024-09-18T15:10:03.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:10:03.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:10:03.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:10:03.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:10:03.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:10:03.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:10:03.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:10:03.114 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:10:03.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:10:03.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:10:03.217 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:10:03.210+0000 7f1e7ded0540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:10:03.228 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:10:03.218+0000 7f1e7ded0540 -1 journal do_read_entry(552960): bad header magic 2024-09-18T15:10:03.228 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:10:03.218+0000 7f1e7ded0540 -1 journal do_read_entry(552960): bad header magic 2024-09-18T15:10:03.236 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:10:03.226+0000 7f1e7ded0540 -1 osd.1 47 log_to_monitors true 2024-09-18T15:10:03.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:10:04.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:10:04.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:10:04.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:10:04.553 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:10:04.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:10:04.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:10:05.030 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 52 up_thru 52 down_at 48 last_clean_interval [10,47) [v2:127.0.0.1:6810/220830,v1:127.0.0.1:6811/220830] [v2:127.0.0.1:6812/220830,v1:127.0.0.1:6813/220830] exists,up 115f2623-03c8-4519-b5cd-6dfb4555432f 2024-09-18T15:10:05.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:10:05.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:10:05.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:10:05.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:10:05.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:10:05.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:10:05.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:10:05.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:10:05.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:10:05.032 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:10:05.032 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:10:05.033 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:10:05.033 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:10:05.033 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:10:05.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:10:05.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:10:05.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:10:05.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:10:05.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:10:05.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:10:05.561 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:10:05.561 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:10:05.561 INFO:tasks.workunit.client.0.smithi046.stderr:3' 2024-09-18T15:10:05.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:10:05.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:10:05.562 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:10:05.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836498 2024-09-18T15:10:05.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836498 2024-09-18T15:10:05.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498' 2024-09-18T15:10:05.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:10:05.703 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:10:05.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=223338299394 2024-09-18T15:10:05.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 223338299394 2024-09-18T15:10:05.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498 1-223338299394' 2024-09-18T15:10:05.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:10:05.838 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:10:05.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509457 2024-09-18T15:10:05.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509457 2024-09-18T15:10:05.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498 1-223338299394 2-64424509457' 2024-09-18T15:10:05.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:10:05.978 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:10:06.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=197568495621 2024-09-18T15:10:06.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 197568495621 2024-09-18T15:10:06.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498 1-223338299394 2-64424509457 3-197568495621' 2024-09-18T15:10:06.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:10:06.119 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836498 2024-09-18T15:10:06.119 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:10:06.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:10:06.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836498 2024-09-18T15:10:06.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:10:06.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836498 2024-09-18T15:10:06.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-09-18T15:10:06.125 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836498 2024-09-18T15:10:06.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:10:06.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836497 -lt 21474836498 2024-09-18T15:10:06.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:10:07.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:10:07.437 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:10:07.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836499 -lt 21474836498 2024-09-18T15:10:07.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:10:07.756 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-223338299394 2024-09-18T15:10:07.756 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:10:07.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:10:07.759 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-223338299394 2024-09-18T15:10:07.759 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:10:07.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=223338299394 2024-09-18T15:10:07.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 223338299394' 2024-09-18T15:10:07.762 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 223338299394 2024-09-18T15:10:07.762 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:10:08.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 223338299394 -lt 223338299394 2024-09-18T15:10:08.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:10:08.093 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509457 2024-09-18T15:10:08.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:10:08.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:10:08.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509457 2024-09-18T15:10:08.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:10:08.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509457 2024-09-18T15:10:08.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-09-18T15:10:08.100 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509457 2024-09-18T15:10:08.101 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:10:08.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509457 -lt 64424509457 2024-09-18T15:10:08.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:10:08.415 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-197568495621 2024-09-18T15:10:08.415 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:10:08.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:10:08.418 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-197568495621 2024-09-18T15:10:08.419 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:10:08.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=197568495621 2024-09-18T15:10:08.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 197568495621' 2024-09-18T15:10:08.420 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.3 seq 197568495621 2024-09-18T15:10:08.420 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:10:08.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 197568495621 -lt 197568495621 2024-09-18T15:10:08.774 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:10:08.775 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:10:08.775 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:10:09.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:10:09.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:10:09.194 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:10:09.194 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:10:09.194 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:10:09.194 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:10:09.194 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:10:09.195 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:10:09.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:10:09.558 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:10:09.559 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:10:09.559 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:10:09.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:10:09.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:10:09.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:10:09.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-18T15:10:09.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-09-18T15:10:09.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T15:10:09.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:10:09.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:10:10.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: local pg=2.0 2024-09-18T15:10:10.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: repair 2.0 2024-09-18T15:10:10.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-09-18T15:10:10.272 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-09-18T15:10:10.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:10:10.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:10:10.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:10:10.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:10:10.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T15:09:45.261277+0000 2024-09-18T15:10:10.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-09-18T15:10:10.910 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-18T15:10:10.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-09-18T15:09:45.261277+0000 2024-09-18T15:10:10.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=2.0 2024-09-18T15:10:10.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:09:45.261277+0000 2024-09-18T15:10:10.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:10:10.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:10:10.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:10:10.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:10:10.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:10:10.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:10:10.924 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:10:10.924 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:10:11.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:09:45.261277+0000 '>' 2024-09-18T15:09:45.261277+0000 2024-09-18T15:10:11.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:10:12.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:10:12.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:10:12.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:10:12.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:10:12.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:10:12.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:10:12.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:10:12.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:09:45.261277+0000 '>' 2024-09-18T15:09:45.261277+0000 2024-09-18T15:10:12.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:10:13.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:10:13.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:10:13.580 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:10:13.580 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:10:13.580 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:10:13.580 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:10:13.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:10:13.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:09:45.261277+0000 '>' 2024-09-18T15:09:45.261277+0000 2024-09-18T15:10:13.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:10:14.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:10:14.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:10:14.897 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:10:14.897 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:10:14.897 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:10:14.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:10:14.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:10:15.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:09:45.261277+0000 '>' 2024-09-18T15:09:45.261277+0000 2024-09-18T15:10:15.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:10:16.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:10:16.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:10:16.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:10:16.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:10:16.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:10:16.209 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:10:16.209 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:10:16.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:09:45.261277+0000 '>' 2024-09-18T15:09:45.261277+0000 2024-09-18T15:10:16.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:10:17.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:10:17.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:10:17.526 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:10:17.526 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:10:17.526 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:10:17.526 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:10:17.527 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:10:17.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:09:45.261277+0000 '>' 2024-09-18T15:09:45.261277+0000 2024-09-18T15:10:17.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:10:18.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:10:18.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:10:18.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:10:18.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:10:18.843 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:10:18.843 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:10:18.843 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:10:19.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:09:45.261277+0000 '>' 2024-09-18T15:09:45.261277+0000 2024-09-18T15:10:19.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:10:20.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:10:20.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:10:20.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:10:20.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:10:20.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:10:20.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:10:20.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:10:20.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:10:13.924093+0000 '>' 2024-09-18T15:09:45.261277+0000 2024-09-18T15:10:20.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:10:20.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:241: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:10:20.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:10:20.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:10:20.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-18T15:10:20.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:10:20.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:10:20.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:10:20.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:10:20.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-18T15:10:20.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:10:20.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-18T15:10:20.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:10:20.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:10:20.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:10:20.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:10:20.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:10:20.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:10:20.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:10:20.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:10:20.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:10:20.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-18T15:10:20.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:10:20.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:10:20.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-18T15:10:21.758 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-18T15:10:21.759 INFO:tasks.workunit.client.0.smithi046.stdout:_ 2024-09-18T15:10:21.759 INFO:tasks.workunit.client.0.smithi046.stdout:hinfo_key 2024-09-18T15:10:21.759 INFO:tasks.workunit.client.0.smithi046.stdout:snapset 2024-09-18T15:10:21.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-18T15:10:21.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:10:21.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:10:21.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T15:10:21.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:10:21.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:10:21.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:10:21.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:10:21.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:10:21.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:10:21.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-18T15:10:21.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-18T15:10:21.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:10:21.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:10:21.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:10:21.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:10:21.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:10:21.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:10:21.819 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:10:21.819 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:10:21.819 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:10:21.819 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:10:21.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:10:21.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:10:21.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:10:21.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:10:21.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:10:21.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:10:21.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:10:21.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:10:21.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:10:21.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:10:21.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-18T15:10:21.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T15:10:21.821 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.3 2024-09-18T15:10:21.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:10:21.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-18T15:10:21.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T15:10:21.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:10:21.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:10:21.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:10:21.853 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:10:21.846+0000 7fa9b4ec5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:10:21.884 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:10:21.874+0000 7fa9b4ec5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:10:21.886 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:10:21.878+0000 7fa9b4ec5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:10:21.888 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:10:21.882+0000 7fa9b4ec5540 -1 Falling back to public interface 2024-09-18T15:10:22.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T15:10:22.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:10:22.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:10:22.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:10:22.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:10:22.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:10:22.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:10:22.574 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:10:22.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:10:22.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:10:22.641 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:10:22.634+0000 7fa9b4ec5540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:10:22.645 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:10:22.638+0000 7fa9b4ec5540 -1 journal do_read_entry(679936): bad header magic 2024-09-18T15:10:22.645 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:10:22.638+0000 7fa9b4ec5540 -1 journal do_read_entry(679936): bad header magic 2024-09-18T15:10:22.675 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:10:22.666+0000 7fa9b4ec5540 -1 osd.3 54 log_to_monitors true 2024-09-18T15:10:22.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:10:23.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:10:23.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:10:23.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:10:23.905 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:10:23.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:10:23.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:10:24.253 INFO:tasks.workunit.client.0.smithi046.stderr:osd.3 up in weight 1 up_from 59 up_thru 59 down_at 55 last_clean_interval [46,54) [v2:127.0.0.1:6826/221767,v1:127.0.0.1:6827/221767] [v2:127.0.0.1:6828/221767,v1:127.0.0.1:6829/221767] exists,up d82b14ff-15f5-422f-9dfe-8c6bfd9425b4 2024-09-18T15:10:24.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:10:24.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:10:24.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:10:24.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:10:24.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:10:24.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:10:24.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:10:24.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:10:24.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:10:24.255 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:10:24.255 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:10:24.255 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:10:24.255 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:10:24.256 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:10:24.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:10:24.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:10:24.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:10:24.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:10:24.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:10:24.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:10:24.801 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:10:24.802 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:10:24.802 INFO:tasks.workunit.client.0.smithi046.stderr:3' 2024-09-18T15:10:24.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:10:24.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:10:24.802 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:10:24.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836503 2024-09-18T15:10:24.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836503 2024-09-18T15:10:24.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836503' 2024-09-18T15:10:24.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:10:24.979 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:10:25.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=223338299399 2024-09-18T15:10:25.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 223338299399 2024-09-18T15:10:25.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836503 1-223338299399' 2024-09-18T15:10:25.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:10:25.116 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:10:25.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509461 2024-09-18T15:10:25.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509461 2024-09-18T15:10:25.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836503 1-223338299399 2-64424509461' 2024-09-18T15:10:25.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:10:25.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:10:25.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=253403070466 2024-09-18T15:10:25.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 253403070466 2024-09-18T15:10:25.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836503 1-223338299399 2-64424509461 3-253403070466' 2024-09-18T15:10:25.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:10:25.403 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836503 2024-09-18T15:10:25.403 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:10:25.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:10:25.406 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836503 2024-09-18T15:10:25.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:10:25.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836503 2024-09-18T15:10:25.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-18T15:10:25.409 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836503 2024-09-18T15:10:25.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:10:25.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836503 -lt 21474836503 2024-09-18T15:10:25.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:10:25.726 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-223338299399 2024-09-18T15:10:25.727 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:10:25.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:10:25.730 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-223338299399 2024-09-18T15:10:25.731 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:10:25.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=223338299399 2024-09-18T15:10:25.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 223338299399' 2024-09-18T15:10:25.732 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 223338299399 2024-09-18T15:10:25.733 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:10:26.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 223338299399 -lt 223338299399 2024-09-18T15:10:26.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:10:26.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509461 2024-09-18T15:10:26.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:10:26.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:10:26.089 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509461 2024-09-18T15:10:26.090 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:10:26.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509461 2024-09-18T15:10:26.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-09-18T15:10:26.092 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509461 2024-09-18T15:10:26.093 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:10:26.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509460 -lt 64424509461 2024-09-18T15:10:26.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:10:27.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:10:27.394 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:10:27.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509462 -lt 64424509461 2024-09-18T15:10:27.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:10:27.718 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-253403070466 2024-09-18T15:10:27.718 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:10:27.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:10:27.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-253403070466 2024-09-18T15:10:27.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:10:27.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=253403070466 2024-09-18T15:10:27.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 253403070466' 2024-09-18T15:10:27.724 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.3 seq 253403070466 2024-09-18T15:10:27.725 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:10:28.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 253403070466 -lt 253403070466 2024-09-18T15:10:28.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:10:28.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:10:28.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:10:28.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:10:28.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:10:28.450 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:10:28.450 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:10:28.450 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:10:28.450 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:10:28.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:10:28.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:10:28.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:10:28.774 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:10:28.774 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:10:28.774 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:10:29.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:10:29.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:10:29.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:10:29.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-18T15:10:29.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-18T15:10:29.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-09-18T15:10:29.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:184: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-18T15:10:29.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local poolname=ecpool 2024-09-18T15:10:29.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local first=1 2024-09-18T15:10:29.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local second=2 2024-09-18T15:10:29.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:192: corrupt_and_repair_two: pids= 2024-09-18T15:10:29.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:10:29.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:10:29.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:10:29.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 222374"' 2024-09-18T15:10:29.255 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 222374' 2024-09-18T15:10:29.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-18T15:10:29.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:10:29.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:10:29.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:10:29.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 222376"' 2024-09-18T15:10:29.255 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 222376' 2024-09-18T15:10:29.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/222377: /' 2024-09-18T15:10:29.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: wait_background pids 2024-09-18T15:10:29.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: wait_background: pids=' 222374 222376' 2024-09-18T15:10:29.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: wait_background: return_code=0 2024-09-18T15:10:29.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:10:29.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 222374 2024-09-18T15:10:29.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-18T15:10:29.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/222379: /' 2024-09-18T15:10:31.094 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:10:31.094 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:10:31.094 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-18T15:10:31.094 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:10:31.094 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-09-18T15:10:31.094 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:10:31.094 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:10:31.094 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-18T15:10:31.094 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:10:31.094 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-18T15:10:31.094 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:10:31.095 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:10:31.095 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:10:31.095 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:10:31.095 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:10:31.095 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:10:31.095 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-09-18T15:10:31.095 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:10:31.095 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:10:31.095 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-18T15:10:31.095 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:10:31.095 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:10:31.095 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-09-18T15:10:31.095 INFO:tasks.workunit.client.0.smithi046.stderr:222379: remove 2#2:eb822e21:::SOMETHING:head# 2024-09-18T15:10:31.096 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-18T15:10:31.096 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:10:31.096 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:10:31.096 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T15:10:31.096 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:10:31.096 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:10:31.096 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:10:31.096 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:10:31.096 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: 222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:10:31.096 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:10:31.096 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:10:31.096 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:10:31.097 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:10:31.097 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:10:31.097 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:10:31.097 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:10:31.097 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:10:31.097 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:10:31.097 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:10:31.097 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:10:31.097 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:10:31.097 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:10:31.097 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:10:31.097 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:10:31.097 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:10:31.098 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:10:31.098 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:10:31.098 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:10:31.098 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:10:31.098 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:10:31.098 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-18T15:10:31.098 INFO:tasks.workunit.client.0.smithi046.stderr:222377: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-18T15:10:31.098 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:10:31.098 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:10:31.098 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:10:31.098 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:10:31.098 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:10:31.098 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:10:31.099 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:10:31.099 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:10:31.100 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:10:31.100 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:10:31.100 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:10:31.100 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:10:31.101 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:10:31.101 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:10:31.101 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:10:31.101 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:10:31.101 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:10:31.101 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:10:31.101 INFO:tasks.workunit.client.0.smithi046.stderr:222377: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:10:31.101 INFO:tasks.workunit.client.0.smithi046.stderr:222377: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:10:31.101 INFO:tasks.workunit.client.0.smithi046.stderr:222377: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:10:31.101 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:10:31.101 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:10:31.101 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:10:31.101 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:10:31.101 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:10:31.102 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:10:31.102 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:10:31.102 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:10:31.102 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:10:31.102 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:10:31.102 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:10:31.102 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:10:31.102 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:10:31.102 INFO:tasks.workunit.client.0.smithi046.stderr:222377: start osd.1 2024-09-18T15:10:31.103 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:10:31.103 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:10:31.103 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T15:10:31.103 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T15:10:31.103 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:10:31.103 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:10:31.103 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:10:31.103 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:10:31.103 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:10:31.103 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:10:31.103 INFO:tasks.workunit.client.0.smithi046.stderr:222379: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:10:31.104 INFO:tasks.workunit.client.0.smithi046.stderr:222379: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:10:31.104 INFO:tasks.workunit.client.0.smithi046.stderr:222379: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:10:31.104 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:10:31.104 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:10:31.104 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:10:31.104 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:10:31.104 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:10:31.104 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:10:31.104 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:10:31.104 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:10:31.104 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:10:31.104 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:10:31.105 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:10:31.105 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-18T15:10:31.105 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T15:10:31.105 INFO:tasks.workunit.client.0.smithi046.stderr:222379: start osd.2 2024-09-18T15:10:33.081 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:10:33.082 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:10:33.082 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:10:33.082 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:10:33.082 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:10:33.082 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:10:33.082 INFO:tasks.workunit.client.0.smithi046.stderr:222377: 2024-09-18T15:10:31.118+0000 7f4875b16540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:10:33.082 INFO:tasks.workunit.client.0.smithi046.stderr:222377: 2024-09-18T15:10:31.126+0000 7f4875b16540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:10:33.082 INFO:tasks.workunit.client.0.smithi046.stderr:222377: 2024-09-18T15:10:31.130+0000 7f4875b16540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:10:33.082 INFO:tasks.workunit.client.0.smithi046.stderr:222377: 2024-09-18T15:10:31.134+0000 7f4875b16540 -1 Falling back to public interface 2024-09-18T15:10:33.082 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:10:33.082 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:10:33.082 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:10:33.082 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:10:33.083 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:10:33.083 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:10:33.083 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:10:33.083 INFO:tasks.workunit.client.0.smithi046.stderr:222377: 0 2024-09-18T15:10:33.083 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:10:33.083 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:10:33.083 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:10:33.083 INFO:tasks.workunit.client.0.smithi046.stderr:222377: 2024-09-18T15:10:31.822+0000 7f4875b16540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:10:33.083 INFO:tasks.workunit.client.0.smithi046.stderr:222377: 2024-09-18T15:10:31.850+0000 7f4875b16540 -1 journal do_read_entry(831488): bad header magic 2024-09-18T15:10:33.083 INFO:tasks.workunit.client.0.smithi046.stderr:222377: 2024-09-18T15:10:31.850+0000 7f4875b16540 -1 journal do_read_entry(831488): bad header magic 2024-09-18T15:10:33.083 INFO:tasks.workunit.client.0.smithi046.stderr:222377: 2024-09-18T15:10:31.862+0000 7f4875b16540 -1 osd.1 60 log_to_monitors true 2024-09-18T15:10:33.083 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:10:33.083 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:10:33.083 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:10:33.083 INFO:tasks.workunit.client.0.smithi046.stderr:222377: 1 2024-09-18T15:10:33.084 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:10:33.084 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:10:33.084 INFO:tasks.workunit.client.0.smithi046.stderr:222377: osd.1 up in weight 1 up_from 66 up_thru 66 down_at 61 last_clean_interval [52,60) [v2:127.0.0.1:6811/222522,v1:127.0.0.1:6813/222522] [v2:127.0.0.1:6815/222522,v1:127.0.0.1:6817/222522] exists,up 115f2623-03c8-4519-b5cd-6dfb4555432f 2024-09-18T15:10:33.084 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:10:33.084 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:10:33.084 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:10:33.084 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:10:33.084 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:10:33.084 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:10:33.085 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_cleops 2024-09-18T15:10:33.085 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-18T15:10:33.085 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T15:10:33.085 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:10:33.085 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:10:33.085 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:10:33.085 INFO:tasks.workunit.client.0.smithi046.stderr:222379: 2024-09-18T15:10:31.118+0000 7f07a2702540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:10:33.085 INFO:tasks.workunit.client.0.smithi046.stderr:222379: 2024-09-18T15:10:31.130+0000 7f07a2702540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:10:33.085 INFO:tasks.workunit.client.0.smithi046.stderr:222379: 2024-09-18T15:10:31.130+0000 7f07a2702540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:10:33.085 INFO:tasks.workunit.client.0.smithi046.stderr:222379: 2024-09-18T15:10:31.134+0000 7f07a2702540 -1 Falling back to public interface 2024-09-18T15:10:33.085 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T15:10:33.085 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:10:33.086 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:10:33.086 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:10:33.086 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:10:33.086 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:10:33.086 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:10:33.086 INFO:tasks.workunit.client.0.smithi046.stderr:222379: 0 2024-09-18T15:10:33.086 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:10:33.086 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:10:33.086 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:10:33.086 INFO:tasks.workunit.client.0.smithi046.stderr:222379: 2024-09-18T15:10:31.790+0000 7f07a2702540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:10:33.086 INFO:tasks.workunit.client.0.smithi046.stderr:222379: 2024-09-18T15:10:31.798+0000 7f07a2702540 -1 journal do_read_entry(786432): bad header magic 2024-09-18T15:10:33.086 INFO:tasks.workunit.client.0.smithi046.stderr:222379: 2024-09-18T15:10:31.798+0000 7f07a2702540 -1 journal do_read_entry(786432): bad header magic 2024-09-18T15:10:33.086 INFO:tasks.workunit.client.0.smithi046.stderr:222379: 2024-09-18T15:10:31.822+0000 7f07a2702540 -1 osd.2 60 log_to_monitors true 2024-09-18T15:10:33.087 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:10:33.087 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:10:33.087 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:10:33.087 INFO:tasks.workunit.client.0.smithi046.stderr:222379: 1 2024-09-18T15:10:33.087 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:10:33.087 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:10:33.087 INFO:tasks.workunit.client.0.smithi046.stderr:222379: osd.2 up in weight 1 up_from 67 up_thru 48 down_at 61 last_clean_interval [15,60) [v2:127.0.0.1:6810/222526,v1:127.0.0.1:6812/222526] [v2:127.0.0.1:6814/222526,v1:127.0.0.1:6816/222526] exists,up 8880f823-a5a9-4299-a2ed-dbe0431c26c7 2024-09-18T15:10:33.087 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:10:33.087 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:10:33.087 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:10:33.087 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:10:33.087 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:10:33.087 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:10:34.513 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:10:34.513 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:10:34.513 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:10:34.513 INFO:tasks.workunit.client.0.smithi046.stderr:222379: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:10:34.514 INFO:tasks.workunit.client.0.smithi046.stderr:222379: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:10:34.514 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:10:34.514 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:10:34.514 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:10:34.514 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:10:34.514 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:10:34.514 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:10:34.514 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:10:34.514 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:10:34.514 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:10:34.514 INFO:tasks.workunit.client.0.smithi046.stderr:222379: 1 2024-09-18T15:10:34.515 INFO:tasks.workunit.client.0.smithi046.stderr:222379: 2 2024-09-18T15:10:34.515 INFO:tasks.workunit.client.0.smithi046.stderr:222379: 3' 2024-09-18T15:10:34.515 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:10:34.515 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:10:34.515 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:10:34.515 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836507 2024-09-18T15:10:34.515 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836507 2024-09-18T15:10:34.515 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507' 2024-09-18T15:10:34.515 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:10:34.516 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:10:34.516 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=283467841539 2024-09-18T15:10:34.516 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 283467841539 2024-09-18T15:10:34.516 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507 1-283467841539' 2024-09-18T15:10:34.516 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:10:34.516 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:10:34.516 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=287762808834 2024-09-18T15:10:34.516 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 287762808834 2024-09-18T15:10:34.516 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507 1-283467841539 2-287762808834' 2024-09-18T15:10:34.516 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:10:34.516 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:10:34.516 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=253403070470 2024-09-18T15:10:34.517 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 253403070470 2024-09-18T15:10:34.517 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:217an: local cur_active_clean 2024-09-18T15:10:34.517 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:10:34.517 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:10:34.517 INFO:tasks.workunit.client.0.smithi046.stderr:222377: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:10:34.517 INFO:tasks.workunit.client.0.smithi046.stderr:222377: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:10:34.517 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:10:34.517 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:10:34.517 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:10:34.517 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:10:34.517 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:10:34.517 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:10:34.518 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:10:34.518 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:10:34.518 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:10:34.518 INFO:tasks.workunit.client.0.smithi046.stderr:222377: 1 2024-09-18T15:10:34.518 INFO:tasks.workunit.client.0.smithi046.stderr:222377: 2 2024-09-18T15:10:34.518 INFO:tasks.workunit.client.0.smithi046.stderr:222377: 3' 2024-09-18T15:10:34.518 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:10:34.518 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:10:34.518 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:10:34.518 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836506 2024-09-18T15:10:34.518 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836506 2024-09-18T15:10:34.518 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836506' 2024-09-18T15:10:34.518 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:10:34.519 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:10:34.519 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=283467841538 2024-09-18T15:10:34.519 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 283467841538 2024-09-18T15:10:34.519 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836506 1-283467841538' 2024-09-18T15:10:34.519 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:10:34.519 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:10:34.519 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=287762808835 2024-09-18T15:10:34.519 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 287762808835 2024-09-18T15:10:34.519 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836506 1-283467841538 2-287762808835' 2024-09-18T15:10:34.519 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:10:34.519 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:10:34.519 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=253403070469 2024-09-18T15:10:34.519 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 253403070469 2024-09-18T15:10:36.449 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836506 1-283467841538 2-287762808835 3-253403070469' 2024-09-18T15:10:36.450 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:10:36.450 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836506 2024-09-18T15:10:36.450 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:10:36.450 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:10:36.450 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836506 2024-09-18T15:10:36.450 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:10:36.450 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836506 2024-09-18T15:10:36.450 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-18T15:10:36.450 INFO:tasks.workunit.client.0.smithi046.stderr:222377: waiting osd.0 seq 21474836506 2024-09-18T15:10:36.450 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:10:36.450 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836505 -lt 21474836506 2024-09-18T15:10:36.451 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:10:36.451 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:10:36.451 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:10:36.451 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836507 -lt 21474836506 2024-09-18T15:10:36.451 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:10:36.451 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-283467841538 2024-09-18T15:10:36.451 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:10:36.451 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:10:36.451 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-283467841538 2024-09-18T15:10:36.451 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:10:36.451 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=283467841538 2024-09-18T15:10:36.451 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 283467841538' 2024-09-18T15:10:36.451 INFO:tasks.workunit.client.0.smithi046.stderr:222377: waiting osd.1 seq 283467841538 2024-09-18T15:10:36.451 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:10:36.452 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 283467841539 -lt 283467841538 2024-09-18T15:10:36.452 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:10:36.452 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-287762808835 2024-09-18T15:10:36.452 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:10:36.452 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:10:36.452 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-287762808835 2024-09-18T15:10:36.452 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:10:36.452 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=287762808835 2024-09-18T15:10:36.452 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 287762808835' 2024-09-18T15:10:36.452 INFO:tasks.workunit.client.0.smithi046.stderr:222377: waiting osd.2 seq 287762808835 2024-09-18T15:10:36.494 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last8: flush_pg_stats: seqs=' 0-21474836507 1-283467841539 2-287762808834 3-253403070470' 2024-09-18T15:10:36.495 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:10:36.495 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836507 2024-09-18T15:10:36.495 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:10:36.495 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:10:36.495 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836507 2024-09-18T15:10:36.495 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:10:36.495 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836507 2024-09-18T15:10:36.495 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-18T15:10:36.495 INFO:tasks.workunit.client.0.smithi046.stderr:222379: waiting osd.0 seq 21474836507 2024-09-18T15:10:36.495 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:10:36.495 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836505 -lt 21474836507 2024-09-18T15:10:36.495 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:10:36.496 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:10:36.496 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:10:36.496 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836507 -lt 21474836507 2024-09-18T15:10:36.496 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:10:36.496 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-283467841539 2024-09-18T15:10:36.496 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:10:36.496 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:10:36.496 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-283467841539 2024-09-18T15:10:36.496 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:10:36.496 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=283467841539 2024-09-18T15:10:36.496 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 283467841539' 2024-09-18T15:10:36.496 INFO:tasks.workunit.client.0.smithi046.stderr:222379: waiting osd.1 seq 283467841539 2024-09-18T15:10:36.496 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:10:36.497 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 283467841539 -lt 283467841539 2024-09-18T15:10:36.497 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:10:36.497 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-287762808834 2024-09-18T15:10:36.497 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:10:36.497 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:10:36.497 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-287762808834 2024-09-18T15:10:36.497 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:10:36.497 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=287762808834 2024-09-18T15:10:36.497 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 287762808834' 2024-09-18T15:10:36.498 INFO:tasks.workunit.client.0.smithi046.stderr:222379: waiting osd.2 seq 287762808834 2024-09-18T15:10:38.334 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:10:38.334 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 287762808835 -lt 287762808834 2024-09-18T15:10:38.334 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:10:38.335 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-253403070470 2024-09-18T15:10:38.335 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:10:38.335 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:10:38.335 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-253403070470 2024-09-18T15:10:38.335 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:10:38.335 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=253403070470 2024-09-18T15:10:38.335 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 253403070470' 2024-09-18T15:10:38.335 INFO:tasks.workunit.client.0.smithi046.stderr:222379: waiting osd.3 seq 253403070470 2024-09-18T15:10:38.335 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:10:38.335 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 253403070470 -lt 253403070470 2024-09-18T15:10:38.335 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:10:38.335 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:10:38.335 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:10:38.335 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:10:38.336 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:10:38.336 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:10:38.336 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:10:38.336 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:10:38.336 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:10:38.336 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:10:38.336 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:10:38.336 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:10:38.336 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:10:38.336 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:10:38.336 INFO:tasks.workunit.client.0.smithi046.stderr:222379: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:10:38.336 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:10:38.336 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:10:38.337 INFO:tasks.workunit.client.0.smithi046.stderr:222379: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:10:38.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:10:38.341 INFO:tasks.workunit.client.0.smithi046.stderr:-stat-seq 2 2024-09-18T15:10:38.341 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 287762808835 -lt 287762808835 2024-09-18T15:10:38.341 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:10:38.341 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-253403070469 2024-09-18T15:10:38.341 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:10:38.342 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:10:38.342 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-253403070469 2024-09-18T15:10:38.342 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:10:38.342 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=253403070469 2024-09-18T15:10:38.342 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 253403070469' 2024-09-18T15:10:38.342 INFO:tasks.workunit.client.0.smithi046.stderr:222377: waiting osd.3 seq 253403070469 2024-09-18T15:10:38.342 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:10:38.342 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 253403070470 -lt 253403070469 2024-09-18T15:10:38.342 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:10:38.342 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:10:38.342 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:10:38.342 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:10:38.342 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:10:38.343 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:10:38.343 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:10:38.343 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:10:38.343 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:10:38.343 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:10:38.343 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:10:38.343 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:10:38.343 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:10:38.343 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:10:38.343 INFO:tasks.workunit.client.0.smithi046.stderr:222377: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:10:38.343 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:10:38.343 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:10:38.343 INFO:tasks.workunit.client.0.smithi046.stderr:222377: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:10:38.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:10:38.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:10:38.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 222376 2024-09-18T15:10:38.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: eval 'pids='\'''\''' 2024-09-18T15:10:38.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: pids= 2024-09-18T15:10:38.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: wait_background: return 0 2024-09-18T15:10:38.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: return_code=0 2024-09-18T15:10:38.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-18T15:10:38.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-18T15:10:38.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-09-18T15:10:38.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T15:10:38.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:10:38.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:10:38.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: local pg=2.0 2024-09-18T15:10:38.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: repair 2.0 2024-09-18T15:10:38.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-09-18T15:10:38.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-09-18T15:10:38.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:10:38.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:10:38.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:10:38.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:10:39.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T15:10:13.924093+0000 2024-09-18T15:10:39.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-09-18T15:10:39.503 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-18T15:10:39.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-09-18T15:10:13.924093+0000 2024-09-18T15:10:39.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=2.0 2024-09-18T15:10:39.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:10:13.924093+0000 2024-09-18T15:10:39.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:10:39.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:10:39.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:10:39.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:10:39.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:10:39.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:10:39.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:10:39.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:10:39.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:10:13.924093+0000 '>' 2024-09-18T15:10:13.924093+0000 2024-09-18T15:10:39.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:10:40.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:10:40.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:10:40.820 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:10:40.820 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:10:40.820 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:10:40.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:10:40.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:10:41.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:10:13.924093+0000 '>' 2024-09-18T15:10:13.924093+0000 2024-09-18T15:10:41.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:10:42.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:10:42.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:10:42.140 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:10:42.140 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:10:42.140 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:10:42.140 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:10:42.141 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:10:42.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:10:13.924093+0000 '>' 2024-09-18T15:10:13.924093+0000 2024-09-18T15:10:42.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:10:43.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:10:43.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:10:43.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:10:43.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:10:43.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:10:43.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:10:43.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:10:43.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:10:13.924093+0000 '>' 2024-09-18T15:10:13.924093+0000 2024-09-18T15:10:43.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:10:44.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:10:44.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:10:44.784 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:10:44.784 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:10:44.784 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:10:44.784 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:10:44.785 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:10:45.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:10:13.924093+0000 '>' 2024-09-18T15:10:13.924093+0000 2024-09-18T15:10:45.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:10:46.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:10:46.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:10:46.101 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:10:46.102 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:10:46.102 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:10:46.102 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:10:46.102 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:10:46.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:10:13.924093+0000 '>' 2024-09-18T15:10:13.924093+0000 2024-09-18T15:10:46.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:10:47.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:10:47.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:10:47.409 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:10:47.409 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:10:47.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:10:47.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:10:47.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:10:47.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:10:13.924093+0000 '>' 2024-09-18T15:10:13.924093+0000 2024-09-18T15:10:47.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:10:48.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:10:48.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:10:48.723 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:10:48.723 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:10:48.723 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:10:48.724 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:10:48.724 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:10:49.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:10:13.924093+0000 '>' 2024-09-18T15:10:13.924093+0000 2024-09-18T15:10:49.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:10:50.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:10:50.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:10:50.035 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:10:50.035 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:10:50.035 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:10:50.035 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:10:50.036 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:10:50.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:10:43.695153+0000 '>' 2024-09-18T15:10:13.924093+0000 2024-09-18T15:10:50.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:10:50.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:207: corrupt_and_repair_two: pids= 2024-09-18T15:10:50.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:10:50.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:10:50.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:10:50.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 224000"' 2024-09-18T15:10:50.343 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 224000' 2024-09-18T15:10:50.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-18T15:10:50.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:10:50.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:10:50.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 224001"' 2024-09-18T15:10:50.344 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 224001' 2024-09-18T15:10:50.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: wait_background pids 2024-09-18T15:10:50.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: wait_background: pids=' 224000 224001' 2024-09-18T15:10:50.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: wait_background: return_code=0 2024-09-18T15:10:50.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:10:50.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 224000 2024-09-18T15:10:50.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:10:50.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/224003: /' 2024-09-18T15:10:50.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-18T15:10:50.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/224005: /' 2024-09-18T15:10:51.654 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:10:51.655 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:10:51.655 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:10:51.655 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:10:51.655 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:10:51.655 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:10:51.655 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:10:51.656 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:10:51.656 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:10:51.656 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:10:51.656 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:10:51.656 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:10:51.656 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:10:51.657 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:10:51.657 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:10:51.657 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:10:51.657 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:10:51.657 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:10:51.657 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:10:51.657 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:10:51.658 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:10:51.658 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:10:51.658 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-18T15:10:51.658 INFO:tasks.workunit.client.0.smithi046.stderr:224003: _ 2024-09-18T15:10:51.658 INFO:tasks.workunit.client.0.smithi046.stderr:224003: hinfo_key 2024-09-18T15:10:51.658 INFO:tasks.workunit.client.0.smithi046.stderr:224003: snapset 2024-09-18T15:10:51.659 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:10:51.659 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:10:51.659 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:10:51.659 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:10:51.659 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:10:51.660 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:10:51.660 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:10:51.660 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:10:51.661 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:10:51.661 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:10:51.661 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:10:51.661 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:10:51.661 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:10:51.662 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:10:51.662 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:10:51.662 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:10:51.662 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:10:51.662 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:10:51.662 INFO:tasks.workunit.client.0.smithi046.stderr:224003: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:10:51.662 INFO:tasks.workunit.client.0.smithi046.stderr:224003: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:10:51.662 INFO:tasks.workunit.client.0.smithi046.stderr:224003: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:10:51.662 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:10:51.663 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:10:51.663 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:10:51.663 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:10:51.663 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:10:51.663 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:10:51.663 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:10:51.663 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:10:51.663 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:10:51.663 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:10:51.663 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:10:51.663 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:10:51.663 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:10:51.664 INFO:tasks.workunit.client.0.smithi046.stderr:224003: start osd.1 2024-09-18T15:10:51.873 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:10:51.873 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:10:51.873 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-18T15:10:51.873 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:10:51.873 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-18T15:10:51.873 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:10:51.873 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:10:51.873 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-18T15:10:51.874 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:10:51.874 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-18T15:10:51.874 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:10:51.874 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:10:51.874 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:10:51.874 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:10:51.874 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:10:51.874 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:10:51.874 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-18T15:10:51.874 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:10:51.874 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:10:51.874 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-18T15:10:51.874 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:10:51.874 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:10:51.875 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-09-18T15:10:51.875 INFO:tasks.workunit.client.0.smithi046.stderr:224005: _ 2024-09-18T15:10:51.875 INFO:tasks.workunit.client.0.smithi046.stderr:224005: hinfo_key 2024-09-18T15:10:51.875 INFO:tasks.workunit.client.0.smithi046.stderr:224005: snapset 2024-09-18T15:10:51.875 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-18T15:10:51.875 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:10:51.875 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:10:51.875 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T15:10:51.875 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:10:51.875 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:10:51.875 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:10:51.875 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:10:51.880 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:10:51.880 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:10:51.880 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T15:10:51.880 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T15:10:51.880 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:10:51.880 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:10:51.880 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:10:51.880 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:10:51.880 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:10:51.880 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:10:51.880 INFO:tasks.workunit.client.0.smithi046.stderr:224005: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:10:51.880 INFO:tasks.workunit.client.0.smithi046.stderr:224005: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:10:51.880 INFO:tasks.workunit.client.0.smithi046.stderr:224005: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:10:51.881 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:10:51.881 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:10:51.881 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:10:51.881 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:10:51.881 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:10:51.881 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:10:51.881 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:10:51.881 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:10:51.881 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:10:51.881 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:10:51.881 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:10:51.881 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-18T15:10:51.882 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T15:10:51.882 INFO:tasks.workunit.client.0.smithi046.stderr:224005: start osd.2 2024-09-18T15:10:53.930 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:10:53.930 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-18T15:10:53.930 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T15:10:53.930 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:10:53.931 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:10:53.931 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:10:53.931 INFO:tasks.workunit.client.0.smithi046.stderr:224005: 2024-09-18T15:10:51.902+0000 7faf9186e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:10:53.931 INFO:tasks.workunit.client.0.smithi046.stderr:224005: 2024-09-18T15:10:51.910+0000 7faf9186e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:10:53.931 INFO:tasks.workunit.client.0.smithi046.stderr:224005: 2024-09-18T15:10:51.914+0000 7faf9186e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:10:53.932 INFO:tasks.workunit.client.0.smithi046.stderr:224005: 2024-09-18T15:10:51.914+0000 7faf9186e540 -1 Falling back to public interface 2024-09-18T15:10:53.932 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T15:10:53.932 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:10:53.932 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:10:53.932 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:10:53.933 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:10:53.933 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:10:53.933 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:10:53.933 INFO:tasks.workunit.client.0.smithi046.stderr:224005: 0 2024-09-18T15:10:53.933 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:10:53.934 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:10:53.934 INFO:tasks.workunit.client.0.smithi046.stderr:224005: 2024-09-18T15:10:52.438+0000 7faf9186e540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:10:53.934 INFO:tasks.workunit.client.0.smithi046.stderr:224005: 2024-09-18T15:10:52.458+0000 7faf9186e540 -1 journal do_read_entry(946176): bad header magic 2024-09-18T15:10:53.934 INFO:tasks.workunit.client.0.smithi046.stderr:224005: 2024-09-18T15:10:52.458+0000 7faf9186e540 -1 journal do_read_entry(946176): bad header magic 2024-09-18T15:10:53.934 INFO:tasks.workunit.client.0.smithi046.stderr:224005: 2024-09-18T15:10:52.466+0000 7faf9186e540 -1 osd.2 69 log_to_monitors true 2024-09-18T15:10:53.935 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:10:53.935 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:10:53.935 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:10:53.935 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:10:53.935 INFO:tasks.workunit.client.0.smithi046.stderr:224005: 1 2024-09-18T15:10:53.936 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:10:53.936 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:10:53.936 INFO:tasks.workunit.client.0.smithi046.stderr:224005: osd.2 up in weight 1 up_from 74 up_thru 74 down_at 70 last_clean_interval [67,69) [v2:127.0.0.1:6818/224186,v1:127.0.0.1:6819/224186] [v2:127.0.0.1:6820/224186,v1:127.0.0.1:6821/224186] exists,up 8880f823-a5a9-4299-a2ed-dbe0431c26c7 2024-09-18T15:10:53.936 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:10:53.936 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:10:53.937 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:10:53.937 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:10:53.937 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:10:53.937 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:10:54.091 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wa_recovery_ops 2024-09-18T15:10:54.091 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:10:54.091 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:10:54.092 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:10:54.092 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:10:54.092 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:10:54.092 INFO:tasks.workunit.client.0.smithi046.stderr:224003: 2024-09-18T15:10:51.686+0000 7f43e5a5b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:10:54.093 INFO:tasks.workunit.client.0.smithi046.stderr:224003: 2024-09-18T15:10:51.710+0000 7f43e5a5b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:10:54.093 INFO:tasks.workunit.client.0.smithi046.stderr:224003: 2024-09-18T15:10:51.714+0000 7f43e5a5b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:10:54.093 INFO:tasks.workunit.client.0.smithi046.stderr:224003: 2024-09-18T15:10:51.718+0000 7f43e5a5b540 -1 Falling back to public interface 2024-09-18T15:10:54.093 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:10:54.093 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:10:54.094 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:10:54.094 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:10:54.094 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:10:54.094 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:10:54.094 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:10:54.095 INFO:tasks.workunit.client.0.smithi046.stderr:224003: 0 2024-09-18T15:10:54.095 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:10:54.095 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:10:54.095 INFO:tasks.workunit.client.0.smithi046.stderr:224003: 2024-09-18T15:10:52.614+0000 7f43e5a5b540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:10:54.095 INFO:tasks.workunit.client.0.smithi046.stderr:224003: 2024-09-18T15:10:52.622+0000 7f43e5a5b540 -1 journal do_read_entry(1015808): bad header magic 2024-09-18T15:10:54.096 INFO:tasks.workunit.client.0.smithi046.stderr:224003: 2024-09-18T15:10:52.622+0000 7f43e5a5b540 -1 journal do_read_entry(1015808): bad header magic 2024-09-18T15:10:54.096 INFO:tasks.workunit.client.0.smithi046.stderr:224003: 2024-09-18T15:10:52.630+0000 7f43e5a5b540 -1 osd.1 69 log_to_monitors true 2024-09-18T15:10:54.096 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:10:54.096 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:10:54.096 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:10:54.097 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:10:54.097 INFO:tasks.workunit.client.0.smithi046.stderr:224003: 1 2024-09-18T15:10:54.097 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:10:54.097 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:10:54.097 INFO:tasks.workunit.client.0.smithi046.stderr:224003: osd.1 up in weight 1 up_from 75 up_thru 67 down_at 70 last_clean_interval [66,69) [v2:127.0.0.1:6810/224132,v1:127.0.0.1:6811/224132] [v2:127.0.0.1:6812/224132,v1:127.0.0.1:6813/224132] exists,up 115f2623-03c8-4519-b5cd-6dfb4555432f 2024-09-18T15:10:54.098 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:10:54.098 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:10:54.098 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:10:54.098 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:10:54.099 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:10:54.099 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:10:55.113 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: it_for_clean: local cur_active_clean 2024-09-18T15:10:55.113 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:10:55.113 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:10:55.113 INFO:tasks.workunit.client.0.smithi046.stderr:224005: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:10:55.113 INFO:tasks.workunit.client.0.smithi046.stderr:224005: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:10:55.114 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:10:55.114 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:10:55.114 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:10:55.114 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:10:55.114 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:10:55.114 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:10:55.114 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:10:55.114 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:10:55.114 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:10:55.114 INFO:tasks.workunit.client.0.smithi046.stderr:224005: 1 2024-09-18T15:10:55.114 INFO:tasks.workunit.client.0.smithi046.stderr:224005: 2 2024-09-18T15:10:55.114 INFO:tasks.workunit.client.0.smithi046.stderr:224005: 3' 2024-09-18T15:10:55.115 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:10:55.115 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:10:55.115 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:10:55.115 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836512 2024-09-18T15:10:55.115 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836512 2024-09-18T15:10:55.115 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836512' 2024-09-18T15:10:55.115 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:10:55.115 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:10:55.115 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=322122547202 2024-09-18T15:10:55.116 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 322122547202 2024-09-18T15:10:55.116 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836512 1-322122547202' 2024-09-18T15:10:55.116 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:10:55.116 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:10:55.116 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=317827579906 2024-09-18T15:10:55.116 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 317827579906 2024-09-18T15:10:55.116 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836512 1-322122547202 2-317827579906' 2024-09-18T15:10:55.116 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:10:55.117 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:10:55.117 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=253403070475 2024-09-18T15:10:55.117 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 253403070475 2024-09-18T15:10:55.233 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpwait_for_clean: local cur_active_clean 2024-09-18T15:10:55.233 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:10:55.233 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:10:55.234 INFO:tasks.workunit.client.0.smithi046.stderr:224003: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:10:55.234 INFO:tasks.workunit.client.0.smithi046.stderr:224003: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:10:55.234 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:10:55.234 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:10:55.234 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:10:55.234 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:10:55.234 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:10:55.234 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:10:55.235 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:10:55.235 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:10:55.235 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:10:55.235 INFO:tasks.workunit.client.0.smithi046.stderr:224003: 1 2024-09-18T15:10:55.235 INFO:tasks.workunit.client.0.smithi046.stderr:224003: 2 2024-09-18T15:10:55.235 INFO:tasks.workunit.client.0.smithi046.stderr:224003: 3' 2024-09-18T15:10:55.235 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:10:55.235 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:10:55.235 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:10:55.235 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836513 2024-09-18T15:10:55.235 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836513 2024-09-18T15:10:55.235 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513' 2024-09-18T15:10:55.235 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:10:55.236 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:10:55.236 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=322122547203 2024-09-18T15:10:55.236 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 322122547203 2024-09-18T15:10:55.236 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513 1-322122547203' 2024-09-18T15:10:55.236 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:10:55.236 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:10:55.236 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=317827579907 2024-09-18T15:10:55.236 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 317827579907 2024-09-18T15:10:55.236 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513 1-322122547203 2-317827579907' 2024-09-18T15:10:55.236 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:10:55.236 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:10:55.236 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=253403070476 2024-09-18T15:10:55.236 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 253403070476 2024-09-18T15:10:56.136 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-heers.sh:2178: flush_pg_stats: seqs=' 0-21474836512 1-322122547202 2-317827579906 3-253403070475' 2024-09-18T15:10:56.136 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:10:56.136 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836512 2024-09-18T15:10:56.136 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:10:56.136 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:10:56.136 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836512 2024-09-18T15:10:56.136 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:10:56.136 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836512 2024-09-18T15:10:56.136 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-09-18T15:10:56.137 INFO:tasks.workunit.client.0.smithi046.stderr:224005: waiting osd.0 seq 21474836512 2024-09-18T15:10:56.137 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:10:56.137 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836513 -lt 21474836512 2024-09-18T15:10:56.137 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:10:56.137 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-322122547202 2024-09-18T15:10:56.137 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:10:56.137 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:10:56.137 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-322122547202 2024-09-18T15:10:56.137 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:10:56.137 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=322122547202 2024-09-18T15:10:56.137 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 322122547202' 2024-09-18T15:10:56.137 INFO:tasks.workunit.client.0.smithi046.stderr:224005: waiting osd.1 seq 322122547202 2024-09-18T15:10:56.137 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:10:56.138 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 322122547203 -lt 322122547202 2024-09-18T15:10:56.138 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:10:56.138 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-317827579906 2024-09-18T15:10:56.138 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:10:56.138 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:10:56.138 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-317827579906 2024-09-18T15:10:56.138 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:10:56.138 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=317827579906 2024-09-18T15:10:56.138 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 317827579906' 2024-09-18T15:10:56.138 INFO:tasks.workunit.client.0.smithi046.stderr:224005: waiting osd.2 seq 317827579906 2024-09-18T15:10:56.138 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:10:56.138 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 317827579907 -lt 317827579906 2024-09-18T15:10:56.138 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:10:56.138 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-253403070475 2024-09-18T15:10:56.287 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pglpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513 1-322122547203 2-317827579907 3-253403070476' 2024-09-18T15:10:56.287 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:10:56.287 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836513 2024-09-18T15:10:56.287 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:10:56.287 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:10:56.287 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836513 2024-09-18T15:10:56.287 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:10:56.287 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836513 2024-09-18T15:10:56.287 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-09-18T15:10:56.287 INFO:tasks.workunit.client.0.smithi046.stderr:224003: waiting osd.0 seq 21474836513 2024-09-18T15:10:56.287 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:10:56.288 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836513 -lt 21474836513 2024-09-18T15:10:56.288 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:10:56.288 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-322122547203 2024-09-18T15:10:56.288 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:10:56.288 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:10:56.288 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-322122547203 2024-09-18T15:10:56.288 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:10:56.288 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=322122547203 2024-09-18T15:10:56.288 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 322122547203' 2024-09-18T15:10:56.288 INFO:tasks.workunit.client.0.smithi046.stderr:224003: waiting osd.1 seq 322122547203 2024-09-18T15:10:56.288 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:10:56.288 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 322122547203 -lt 322122547203 2024-09-18T15:10:56.288 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:10:56.289 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-317827579907 2024-09-18T15:10:56.289 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:10:56.289 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:10:56.289 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-317827579907 2024-09-18T15:10:56.289 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:10:56.289 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=317827579907 2024-09-18T15:10:56.289 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 317827579907' 2024-09-18T15:10:56.289 INFO:tasks.workunit.client.0.smithi046.stderr:224003: waiting osd.2 seq 317827579907 2024-09-18T15:10:56.289 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:10:56.289 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 317827579907 -lt 317827579907 2024-09-18T15:10:56.289 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:10:56.289 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-253403070476 2024-09-18T15:10:57.615 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush__stats: cut -d - -f 1 2024-09-18T15:10:57.615 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:10:57.616 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:10:57.616 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-253403070475 2024-09-18T15:10:57.616 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=253403070475 2024-09-18T15:10:57.616 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 253403070475' 2024-09-18T15:10:57.616 INFO:tasks.workunit.client.0.smithi046.stderr:224005: waiting osd.3 seq 253403070475 2024-09-18T15:10:57.616 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:10:57.616 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 253403070476 -lt 253403070475 2024-09-18T15:10:57.616 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:10:57.616 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:10:57.616 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:10:57.616 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:10:57.616 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:10:57.617 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:10:57.617 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:10:57.617 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:10:57.617 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:10:57.617 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:10:57.617 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:10:57.617 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:10:57.617 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:10:57.617 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:10:57.617 INFO:tasks.workunit.client.0.smithi046.stderr:224005: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:10:57.617 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:10:57.617 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:10:57.618 INFO:tasks.workunit.client.0.smithi046.stderr:224005: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:10:57.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:10:57.766 INFO:tasks.workunit.client.0.smithi046.stderr:pg_stats: cut -d - -f 1 2024-09-18T15:10:57.767 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:10:57.767 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-253403070476 2024-09-18T15:10:57.767 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:10:57.767 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=253403070476 2024-09-18T15:10:57.767 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 253403070476' 2024-09-18T15:10:57.767 INFO:tasks.workunit.client.0.smithi046.stderr:224003: waiting osd.3 seq 253403070476 2024-09-18T15:10:57.767 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:10:57.767 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 253403070476 -lt 253403070476 2024-09-18T15:10:57.767 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:10:57.768 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:10:57.768 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:10:57.768 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:10:57.768 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:10:57.768 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:10:57.768 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:10:57.768 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:10:57.768 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:10:57.768 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:10:57.768 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:10:57.768 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:10:57.768 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:10:57.769 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:10:57.769 INFO:tasks.workunit.client.0.smithi046.stderr:224003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:10:57.769 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:10:57.769 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:10:57.769 INFO:tasks.workunit.client.0.smithi046.stderr:224003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:10:57.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:10:57.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:10:57.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 224001 2024-09-18T15:10:57.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: eval 'pids='\'''\''' 2024-09-18T15:10:57.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: pids= 2024-09-18T15:10:57.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: wait_background: return 0 2024-09-18T15:10:57.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: return_code=0 2024-09-18T15:10:57.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-18T15:10:57.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:214: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-18T15:10:57.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-18T15:10:57.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-09-18T15:10:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:184: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-18T15:10:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local poolname=ecpool 2024-09-18T15:10:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local first=3 2024-09-18T15:10:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local second=1 2024-09-18T15:10:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:192: corrupt_and_repair_two: pids= 2024-09-18T15:10:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:10:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:10:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:10:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 225293"' 2024-09-18T15:10:57.969 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 225293' 2024-09-18T15:10:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:10:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:10:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:10:57.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/225295: /' 2024-09-18T15:10:57.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 225296"' 2024-09-18T15:10:57.971 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 225296' 2024-09-18T15:10:57.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: wait_background pids 2024-09-18T15:10:57.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: wait_background: pids=' 225293 225296' 2024-09-18T15:10:57.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: wait_background: return_code=0 2024-09-18T15:10:57.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:10:57.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 225293 2024-09-18T15:10:57.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:10:57.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:10:57.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/225298: /' 2024-09-18T15:10:59.927 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:10:59.928 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:10:59.928 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-18T15:10:59.928 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:10:59.928 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:10:59.928 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:10:59.928 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:10:59.928 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-18T15:10:59.928 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:10:59.928 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-18T15:10:59.929 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:10:59.929 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:10:59.929 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:10:59.929 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:10:59.929 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:10:59.929 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:10:59.929 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:10:59.929 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:10:59.929 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:10:59.929 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-18T15:10:59.929 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:10:59.929 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:10:59.929 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-18T15:10:59.929 INFO:tasks.workunit.client.0.smithi046.stderr:225295: remove 0#2:eb822e21:::SOMETHING:head# 2024-09-18T15:10:59.930 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-18T15:10:59.930 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:10:59.930 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:10:59.930 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T15:10:59.930 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:10:59.930 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:10:59.930 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:10:59.930 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:10:59.934 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:10:59.934 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:10:59.934 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-18T15:10:59.934 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-18T15:10:59.934 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:10:59.935 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:10:59.935 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:10:59.935 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:10:59.935 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:10:59.935 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:10:59.935 INFO:tasks.workunit.client.0.smithi046.stderr:225295: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:10:59.935 INFO:tasks.workunit.client.0.smithi046.stderr:225295: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:10:59.935 INFO:tasks.workunit.client.0.smithi046.stderr:225295: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:10:59.935 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:10:59.935 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:10:59.935 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:10:59.935 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:10:59.935 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:10:59.936 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:10:59.936 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:10:59.936 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:10:59.936 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:10:59.936 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:10:59.936 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:10:59.936 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-18T15:10:59.936 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T15:10:59.936 INFO:tasks.workunit.client.0.smithi046.stderr:225295: start osd.3 2024-09-18T15:11:00.467 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:11:00.468 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:11:00.468 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:11:00.468 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:11:00.468 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:11:00.468 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:11:00.468 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:11:00.468 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:11:00.468 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:11:00.468 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:11:00.469 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:11:00.469 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:11:00.469 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:11:00.469 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:11:00.469 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:11:00.469 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:11:00.469 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:11:00.469 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:11:00.469 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:11:00.469 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:11:00.469 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:11:00.469 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:11:00.469 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-18T15:11:00.470 INFO:tasks.workunit.client.0.smithi046.stderr:225298: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-18T15:11:00.470 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:11:00.470 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:11:00.470 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:11:00.470 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:11:00.470 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:11:00.470 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:11:00.470 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:11:00.470 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:11:00.473 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:11:00.473 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:11:00.473 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:11:00.473 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:11:00.474 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:11:00.474 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:11:00.474 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:11:00.474 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:11:00.474 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:11:00.474 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:11:00.474 INFO:tasks.workunit.client.0.smithi046.stderr:225298: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:11:00.474 INFO:tasks.workunit.client.0.smithi046.stderr:225298: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:11:00.474 INFO:tasks.workunit.client.0.smithi046.stderr:225298: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:11:00.474 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:11:00.474 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:11:00.474 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:11:00.474 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:11:00.475 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:11:00.475 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:11:00.475 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:11:00.475 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:11:00.475 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:11:00.475 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:11:00.475 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:11:00.476 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:11:00.476 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:11:00.476 INFO:tasks.workunit.client.0.smithi046.stderr:225298: start osd.1 2024-09-18T15:11:02.334 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:11:02.334 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-18T15:11:02.334 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T15:11:02.334 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:11:02.334 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:11:02.334 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:11:02.334 INFO:tasks.workunit.client.0.smithi046.stderr:225295: 2024-09-18T15:10:59.958+0000 7f1a7c4a4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:02.335 INFO:tasks.workunit.client.0.smithi046.stderr:225295: 2024-09-18T15:10:59.962+0000 7f1a7c4a4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:02.335 INFO:tasks.workunit.client.0.smithi046.stderr:225295: 2024-09-18T15:10:59.962+0000 7f1a7c4a4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:02.335 INFO:tasks.workunit.client.0.smithi046.stderr:225295: 2024-09-18T15:10:59.966+0000 7f1a7c4a4540 -1 Falling back to public interface 2024-09-18T15:11:02.335 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T15:11:02.335 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:11:02.335 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:11:02.335 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:11:02.335 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:11:02.335 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:02.335 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:11:02.335 INFO:tasks.workunit.client.0.smithi046.stderr:225295: 0 2024-09-18T15:11:02.335 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:02.335 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:11:02.335 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:02.336 INFO:tasks.workunit.client.0.smithi046.stderr:225295: 2024-09-18T15:11:00.886+0000 7f1a7c4a4540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:11:02.336 INFO:tasks.workunit.client.0.smithi046.stderr:225295: 2024-09-18T15:11:00.906+0000 7f1a7c4a4540 -1 journal do_read_entry(1175552): bad header magic 2024-09-18T15:11:02.336 INFO:tasks.workunit.client.0.smithi046.stderr:225295: 2024-09-18T15:11:00.906+0000 7f1a7c4a4540 -1 journal do_read_entry(1175552): bad header magic 2024-09-18T15:11:02.336 INFO:tasks.workunit.client.0.smithi046.stderr:225295: 2024-09-18T15:11:00.930+0000 7f1a7c4a4540 -1 osd.3 77 log_to_monitors true 2024-09-18T15:11:02.336 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:02.336 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:02.336 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:11:02.336 INFO:tasks.workunit.client.0.smithi046.stderr:225295: 1 2024-09-18T15:11:02.336 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:02.336 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:11:02.336 INFO:tasks.workunit.client.0.smithi046.stderr:225295: osd.3 up in weight 1 up_from 82 up_thru 83 down_at 78 last_clean_interval [59,77) [v2:127.0.0.1:6810/225425,v1:127.0.0.1:6811/225425] [v2:127.0.0.1:6812/225425,v1:127.0.0.1:6813/225425] exists,up d82b14ff-15f5-422f-9dfe-8c6bfd9425b4 2024-09-18T15:11:02.336 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:11:02.336 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:11:02.336 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:11:02.336 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:11:02.337 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:11:02.337 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:11:02.560 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_cops 2024-09-18T15:11:02.560 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:11:02.560 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:11:02.560 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:11:02.560 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:11:02.560 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:11:02.560 INFO:tasks.workunit.client.0.smithi046.stderr:225298: 2024-09-18T15:11:00.494+0000 7fd73138c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:02.560 INFO:tasks.workunit.client.0.smithi046.stderr:225298: 2024-09-18T15:11:00.518+0000 7fd73138c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:02.561 INFO:tasks.workunit.client.0.smithi046.stderr:225298: 2024-09-18T15:11:00.522+0000 7fd73138c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:02.561 INFO:tasks.workunit.client.0.smithi046.stderr:225298: 2024-09-18T15:11:00.522+0000 7fd73138c540 -1 Falling back to public interface 2024-09-18T15:11:02.561 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:11:02.561 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:11:02.561 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:11:02.561 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:11:02.561 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:11:02.561 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:02.561 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:11:02.561 INFO:tasks.workunit.client.0.smithi046.stderr:225298: 0 2024-09-18T15:11:02.561 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:02.562 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:11:02.562 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:02.562 INFO:tasks.workunit.client.0.smithi046.stderr:225298: 2024-09-18T15:11:01.166+0000 7fd73138c540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:11:02.562 INFO:tasks.workunit.client.0.smithi046.stderr:225298: 2024-09-18T15:11:01.170+0000 7fd73138c540 -1 journal do_read_entry(1171456): bad header magic 2024-09-18T15:11:02.562 INFO:tasks.workunit.client.0.smithi046.stderr:225298: 2024-09-18T15:11:01.170+0000 7fd73138c540 -1 journal do_read_entry(1171456): bad header magic 2024-09-18T15:11:02.562 INFO:tasks.workunit.client.0.smithi046.stderr:225298: 2024-09-18T15:11:01.182+0000 7fd73138c540 -1 osd.1 77 log_to_monitors true 2024-09-18T15:11:02.562 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:02.562 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:02.562 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:11:02.562 INFO:tasks.workunit.client.0.smithi046.stderr:225298: 1 2024-09-18T15:11:02.562 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:11:02.562 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:02.562 INFO:tasks.workunit.client.0.smithi046.stderr:225298: osd.1 up in weight 1 up_from 83 up_thru 75 down_at 78 last_clean_interval [75,77) [v2:127.0.0.1:6826/225509,v1:127.0.0.1:6827/225509] [v2:127.0.0.1:6828/225509,v1:127.0.0.1:6829/225509] exists,up 115f2623-03c8-4519-b5cd-6dfb4555432f 2024-09-18T15:11:02.563 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:11:02.563 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:11:02.563 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:11:02.563 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:11:02.563 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:11:02.563 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:11:03.635 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:11:03.635 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:11:03.635 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:11:03.635 INFO:tasks.workunit.client.0.smithi046.stderr:225295: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:11:03.635 INFO:tasks.workunit.client.0.smithi046.stderr:225295: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:11:03.635 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:11:03.635 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:11:03.635 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:11:03.635 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:11:03.636 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:11:03.636 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:11:03.636 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:11:03.636 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:11:03.636 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:11:03.636 INFO:tasks.workunit.client.0.smithi046.stderr:225295: 1 2024-09-18T15:11:03.636 INFO:tasks.workunit.client.0.smithi046.stderr:225295: 2 2024-09-18T15:11:03.636 INFO:tasks.workunit.client.0.smithi046.stderr:225295: 3' 2024-09-18T15:11:03.636 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:11:03.636 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:11:03.636 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:11:03.636 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836516 2024-09-18T15:11:03.636 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836516 2024-09-18T15:11:03.636 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836516' 2024-09-18T15:11:03.637 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:11:03.637 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:11:03.637 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=356482285570 2024-09-18T15:11:03.637 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 356482285570 2024-09-18T15:11:03.637 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836516 1-356482285570' 2024-09-18T15:11:03.637 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:11:03.637 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:11:03.637 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=317827579910 2024-09-18T15:11:03.637 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 317827579910 2024-09-18T15:11:03.637 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836516 1-356482285570 2-317827579910' 2024-09-18T15:11:03.637 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:11:03.637 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:11:03.638 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=352187318274 2024-09-18T15:11:03.638 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 352187318274 2024-09-18T15:11:03.782 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2lean: local cur_active_clean 2024-09-18T15:11:03.782 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:11:03.783 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:11:03.783 INFO:tasks.workunit.client.0.smithi046.stderr:225298: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:11:03.783 INFO:tasks.workunit.client.0.smithi046.stderr:225298: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:11:03.783 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:11:03.783 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:11:03.783 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:11:03.783 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:11:03.783 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:11:03.783 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:11:03.783 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:11:03.783 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:11:03.783 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:11:03.784 INFO:tasks.workunit.client.0.smithi046.stderr:225298: 1 2024-09-18T15:11:03.784 INFO:tasks.workunit.client.0.smithi046.stderr:225298: 2 2024-09-18T15:11:03.784 INFO:tasks.workunit.client.0.smithi046.stderr:225298: 3' 2024-09-18T15:11:03.784 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:11:03.784 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:11:03.784 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:11:03.784 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836517 2024-09-18T15:11:03.784 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836517 2024-09-18T15:11:03.784 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836517' 2024-09-18T15:11:03.784 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:11:03.785 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:11:03.785 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=356482285571 2024-09-18T15:11:03.785 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 356482285571 2024-09-18T15:11:03.785 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836517 1-356482285571' 2024-09-18T15:11:03.785 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:11:03.785 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:11:03.785 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=317827579911 2024-09-18T15:11:03.785 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 317827579911 2024-09-18T15:11:03.785 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836517 1-356482285571 2-317827579911' 2024-09-18T15:11:03.785 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:11:03.785 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:11:03.785 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=352187318275 2024-09-18T15:11:03.785 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 352187318275 2024-09-18T15:11:05.597 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836516 1-356482285570 2-317827579910 3-352187318274' 2024-09-18T15:11:05.597 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:11:05.597 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836516 2024-09-18T15:11:05.597 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:11:05.597 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:11:05.597 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836516 2024-09-18T15:11:05.598 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:11:05.598 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836516 2024-09-18T15:11:05.598 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-09-18T15:11:05.598 INFO:tasks.workunit.client.0.smithi046.stderr:225295: waiting osd.0 seq 21474836516 2024-09-18T15:11:05.598 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:11:05.598 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836516 -lt 21474836516 2024-09-18T15:11:05.598 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:11:05.598 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-356482285570 2024-09-18T15:11:05.598 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:11:05.598 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:11:05.598 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-356482285570 2024-09-18T15:11:05.598 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:11:05.598 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=356482285570 2024-09-18T15:11:05.599 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 356482285570' 2024-09-18T15:11:05.599 INFO:tasks.workunit.client.0.smithi046.stderr:225295: waiting osd.1 seq 356482285570 2024-09-18T15:11:05.599 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:11:05.599 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 356482285570 2024-09-18T15:11:05.599 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:11:05.599 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:11:05.599 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:11:05.599 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 356482285571 -lt 356482285570 2024-09-18T15:11:05.599 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:11:05.599 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-317827579910 2024-09-18T15:11:05.599 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:11:05.599 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:11:05.599 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:11:05.599 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-317827579910 2024-09-18T15:11:05.600 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=317827579910 2024-09-18T15:11:05.600 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 317827579910' 2024-09-18T15:11:05.600 INFO:tasks.workunit.client.0.smithi046.stderr:225295: waiting osd.2 seq 317827579910 2024-09-18T15:11:05.788 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-s178: flush_pg_stats: seqs=' 0-21474836517 1-356482285571 2-317827579911 3-352187318275' 2024-09-18T15:11:05.789 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:11:05.789 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836517 2024-09-18T15:11:05.789 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:11:05.789 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:11:05.789 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836517 2024-09-18T15:11:05.789 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:11:05.789 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836517 2024-09-18T15:11:05.789 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2024-09-18T15:11:05.789 INFO:tasks.workunit.client.0.smithi046.stderr:225298: waiting osd.0 seq 21474836517 2024-09-18T15:11:05.789 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:11:05.789 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836516 -lt 21474836517 2024-09-18T15:11:05.789 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:11:05.790 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:11:05.790 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:11:05.790 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836517 -lt 21474836517 2024-09-18T15:11:05.790 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:11:05.790 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:11:05.790 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-356482285571 2024-09-18T15:11:05.790 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:11:05.790 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-356482285571 2024-09-18T15:11:05.790 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:11:05.790 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=356482285571 2024-09-18T15:11:05.790 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 356482285571' 2024-09-18T15:11:05.790 INFO:tasks.workunit.client.0.smithi046.stderr:225298: waiting osd.1 seq 356482285571 2024-09-18T15:11:05.790 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:11:05.790 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 356482285571 -lt 356482285571 2024-09-18T15:11:05.791 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:11:05.791 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-317827579911 2024-09-18T15:11:05.791 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:11:05.791 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:11:05.791 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-317827579911 2024-09-18T15:11:05.791 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:11:05.791 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=317827579911 2024-09-18T15:11:05.791 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 317827579911' 2024-09-18T15:11:05.791 INFO:tasks.workunit.client.0.smithi046.stderr:225298: waiting osd.2 seq 317827579911 2024-09-18T15:11:07.520 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:11:07.521 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 317827579911 -lt 317827579911 2024-09-18T15:11:07.521 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:11:07.521 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-352187318275 2024-09-18T15:11:07.521 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:11:07.521 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:11:07.521 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:11:07.521 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-352187318275 2024-09-18T15:11:07.521 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=352187318275 2024-09-18T15:11:07.521 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 352187318275' 2024-09-18T15:11:07.521 INFO:tasks.workunit.client.0.smithi046.stderr:225298: waiting osd.3 seq 352187318275 2024-09-18T15:11:07.521 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:11:07.521 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 352187318275 -lt 352187318275 2024-09-18T15:11:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:11:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:11:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:11:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:11:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:11:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:11:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:11:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:11:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:11:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:11:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:11:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:11:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:11:07.523 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:11:07.523 INFO:tasks.workunit.client.0.smithi046.stderr:225298: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:11:07.523 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:11:07.523 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:11:07.523 INFO:tasks.workunit.client.0.smithi046.stderr:225298: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:11:07.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:11:07.558 INFO:tasks.workunit.client.0.smithi046.stderr:eq 2 2024-09-18T15:11:07.559 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 317827579911 -lt 317827579910 2024-09-18T15:11:07.559 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:11:07.559 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-352187318274 2024-09-18T15:11:07.559 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:11:07.559 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:11:07.559 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-352187318274 2024-09-18T15:11:07.559 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:11:07.559 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=352187318274 2024-09-18T15:11:07.559 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 352187318274' 2024-09-18T15:11:07.559 INFO:tasks.workunit.client.0.smithi046.stderr:225295: waiting osd.3 seq 352187318274 2024-09-18T15:11:07.559 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:11:07.560 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 352187318275 -lt 352187318274 2024-09-18T15:11:07.560 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:11:07.560 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:11:07.560 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:11:07.560 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:11:07.560 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:11:07.560 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:11:07.560 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:11:07.560 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:11:07.560 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:11:07.560 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:11:07.560 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:11:07.561 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:11:07.561 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:11:07.561 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:11:07.561 INFO:tasks.workunit.client.0.smithi046.stderr:225295: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:11:07.561 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:11:07.561 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:11:07.561 INFO:tasks.workunit.client.0.smithi046.stderr:225295: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:11:07.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:11:07.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:11:07.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 225296 2024-09-18T15:11:07.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: eval 'pids='\'''\''' 2024-09-18T15:11:07.563 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: pids= 2024-09-18T15:11:07.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: wait_background: return 0 2024-09-18T15:11:07.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: return_code=0 2024-09-18T15:11:07.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-18T15:11:07.563 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-18T15:11:07.563 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-09-18T15:11:07.563 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T15:11:07.563 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:11:07.563 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:11:07.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: local pg=2.0 2024-09-18T15:11:07.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: repair 2.0 2024-09-18T15:11:07.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-09-18T15:11:07.862 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-09-18T15:11:07.862 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:11:07.862 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:11:07.862 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:11:07.862 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:11:08.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T15:10:43.695153+0000 2024-09-18T15:11:08.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-09-18T15:11:08.497 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-18T15:11:08.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-09-18T15:10:43.695153+0000 2024-09-18T15:11:08.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=2.0 2024-09-18T15:11:08.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:10:43.695153+0000 2024-09-18T15:11:08.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:11:08.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:11:08.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:11:08.506 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:11:08.506 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:11:08.506 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:11:08.506 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:11:08.507 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:11:08.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:10:43.695153+0000 '>' 2024-09-18T15:10:43.695153+0000 2024-09-18T15:11:08.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:11:09.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:11:09.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:11:09.809 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:11:09.809 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:11:09.809 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:11:09.809 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:11:09.810 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:11:10.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:10:43.695153+0000 '>' 2024-09-18T15:10:43.695153+0000 2024-09-18T15:11:10.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:11:11.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:11:11.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:11:11.125 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:11:11.125 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:11:11.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:11:11.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:11:11.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:11:11.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:10:43.695153+0000 '>' 2024-09-18T15:10:43.695153+0000 2024-09-18T15:11:11.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:11:12.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:11:12.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:11:12.450 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:11:12.450 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:11:12.450 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:11:12.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:11:12.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:11:12.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:10:43.695153+0000 '>' 2024-09-18T15:10:43.695153+0000 2024-09-18T15:11:12.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:11:13.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:11:13.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:11:13.758 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:11:13.758 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:11:13.758 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:11:13.758 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:11:13.759 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:11:14.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:10:43.695153+0000 '>' 2024-09-18T15:10:43.695153+0000 2024-09-18T15:11:14.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:11:15.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:11:15.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:11:15.084 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:11:15.084 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:11:15.084 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:11:15.084 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:11:15.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:11:15.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:10:43.695153+0000 '>' 2024-09-18T15:10:43.695153+0000 2024-09-18T15:11:15.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:11:16.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:11:16.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:11:16.399 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:11:16.399 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:11:16.399 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:11:16.399 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:11:16.400 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:11:16.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:10:43.695153+0000 '>' 2024-09-18T15:10:43.695153+0000 2024-09-18T15:11:16.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:11:17.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:11:17.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:11:17.712 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:11:17.712 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:11:17.712 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:11:17.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:11:17.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:11:18.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:11:16.066480+0000 '>' 2024-09-18T15:10:43.695153+0000 2024-09-18T15:11:18.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:11:18.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:207: corrupt_and_repair_two: pids= 2024-09-18T15:11:18.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:11:18.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:11:18.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:11:18.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 226893"' 2024-09-18T15:11:18.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 226893' 2024-09-18T15:11:18.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:11:18.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:11:18.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:11:18.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:11:18.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 226895"' 2024-09-18T15:11:18.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 226895' 2024-09-18T15:11:18.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/226896: /' 2024-09-18T15:11:18.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: wait_background pids 2024-09-18T15:11:18.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: wait_background: pids=' 226893 226895' 2024-09-18T15:11:18.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: wait_background: return_code=0 2024-09-18T15:11:18.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:11:18.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 226893 2024-09-18T15:11:18.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:11:18.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/226899: /' 2024-09-18T15:11:20.733 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:11:20.733 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:11:20.733 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:11:20.733 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:11:20.733 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:11:20.733 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:11:20.733 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:11:20.733 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:11:20.733 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:11:20.734 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:11:20.734 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:11:20.734 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:11:20.734 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:11:20.734 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:11:20.734 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:11:20.734 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:11:20.734 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:11:20.734 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:11:20.734 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:11:20.734 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:11:20.734 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:11:20.734 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:11:20.735 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-18T15:11:20.735 INFO:tasks.workunit.client.0.smithi046.stderr:226899: _ 2024-09-18T15:11:20.735 INFO:tasks.workunit.client.0.smithi046.stderr:226899: hinfo_key 2024-09-18T15:11:20.735 INFO:tasks.workunit.client.0.smithi046.stderr:226899: snapset 2024-09-18T15:11:20.735 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:11:20.735 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:11:20.735 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:11:20.735 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:11:20.735 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:11:20.735 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:11:20.735 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:11:20.735 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:11:20.736 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:11:20.736 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:11:20.736 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-18T15:11:20.736 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:11:20.736 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:11:20.736 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:11:20.736 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:11:20.736 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-18T15:11:20.736 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:11:20.736 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-18T15:11:20.737 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:11:20.737 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:11:20.737 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:11:20.737 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:11:20.737 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:11:20.737 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:11:20.737 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:11:20.737 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:11:20.737 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:11:20.737 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-18T15:11:20.737 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:11:20.738 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:11:20.738 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-18T15:11:20.738 INFO:tasks.workunit.client.0.smithi046.stderr:226896: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-18T15:11:20.738 INFO:tasks.workunit.client.0.smithi046.stderr:226896: _ 2024-09-18T15:11:20.738 INFO:tasks.workunit.client.0.smithi046.stderr:226896: hinfo_key 2024-09-18T15:11:20.738 INFO:tasks.workunit.client.0.smithi046.stderr:226896: snapset 2024-09-18T15:11:20.738 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-18T15:11:20.738 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:11:20.738 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:11:20.738 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T15:11:20.738 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:11:20.738 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:11:20.738 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:11:20.740 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' s.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:11:20.740 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:11:20.740 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:11:20.741 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:11:20.741 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:11:20.741 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:11:20.741 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:11:20.741 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:11:20.741 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:11:20.741 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:11:20.741 INFO:tasks.workunit.client.0.smithi046.stderr:226899: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:11:20.741 INFO:tasks.workunit.client.0.smithi046.stderr:226899: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:11:20.741 INFO:tasks.workunit.client.0.smithi046.stderr:226899: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:11:20.741 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:11:20.741 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:11:20.741 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:11:20.742 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:11:20.742 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:11:20.742 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:11:20.742 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:11:20.742 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:11:20.742 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:11:20.742 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:11:20.742 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:11:20.742 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:11:20.742 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:11:20.742 INFO:tasks.workunit.client.0.smithi046.stderr:226899: start osd.1 2024-09-18T15:11:20.743 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high--osd-failsafe-full-ratio=.99' 2024-09-18T15:11:20.743 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:11:20.743 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:11:20.743 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-18T15:11:20.743 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-18T15:11:20.743 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:11:20.743 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:11:20.743 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:11:20.743 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:11:20.743 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:11:20.743 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:11:20.744 INFO:tasks.workunit.client.0.smithi046.stderr:226896: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:11:20.744 INFO:tasks.workunit.client.0.smithi046.stderr:226896: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:11:20.744 INFO:tasks.workunit.client.0.smithi046.stderr:226896: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:11:20.744 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:11:20.744 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:11:20.744 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:11:20.744 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:11:20.744 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:11:20.744 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:11:20.744 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:11:20.744 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:11:20.744 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:11:20.745 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:11:20.745 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:11:20.745 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-18T15:11:20.745 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T15:11:20.745 INFO:tasks.workunit.client.0.smithi046.stderr:226896: start osd.3 2024-09-18T15:11:22.835 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:11:22.835 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-18T15:11:22.835 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T15:11:22.835 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:11:22.835 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:11:22.835 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:11:22.836 INFO:tasks.workunit.client.0.smithi046.stderr:226896: 2024-09-18T15:11:20.777+0000 7f515a942540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:22.836 INFO:tasks.workunit.client.0.smithi046.stderr:226896: 2024-09-18T15:11:20.781+0000 7f515a942540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:22.836 INFO:tasks.workunit.client.0.smithi046.stderr:226896: 2024-09-18T15:11:20.785+0000 7f515a942540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:22.836 INFO:tasks.workunit.client.0.smithi046.stderr:226896: 2024-09-18T15:11:20.789+0000 7f515a942540 -1 Falling back to public interface 2024-09-18T15:11:22.836 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T15:11:22.836 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:11:22.836 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:11:22.836 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:11:22.836 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:11:22.836 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:22.836 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:11:22.836 INFO:tasks.workunit.client.0.smithi046.stderr:226896: 0 2024-09-18T15:11:22.836 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:22.836 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:11:22.837 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:22.837 INFO:tasks.workunit.client.0.smithi046.stderr:226896: 2024-09-18T15:11:21.585+0000 7f515a942540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:11:22.837 INFO:tasks.workunit.client.0.smithi046.stderr:226896: 2024-09-18T15:11:21.593+0000 7f515a942540 -1 journal do_read_entry(1331200): bad header magic 2024-09-18T15:11:22.837 INFO:tasks.workunit.client.0.smithi046.stderr:226896: 2024-09-18T15:11:21.593+0000 7f515a942540 -1 journal do_read_entry(1331200): bad header magic 2024-09-18T15:11:22.837 INFO:tasks.workunit.client.0.smithi046.stderr:226896: 2024-09-18T15:11:21.605+0000 7f515a942540 -1 osd.3 85 log_to_monitors true 2024-09-18T15:11:22.837 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:22.837 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:22.837 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:11:22.837 INFO:tasks.workunit.client.0.smithi046.stderr:226896: 1 2024-09-18T15:11:22.837 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:22.837 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:11:22.837 INFO:tasks.workunit.client.0.smithi046.stderr:226896: osd.3 up in weight 1 up_from 92 up_thru 83 down_at 86 last_clean_interval [82,85) [v2:127.0.0.1:6826/227051,v1:127.0.0.1:6827/227051] [v2:127.0.0.1:6828/227051,v1:127.0.0.1:6829/227051] exists,up d82b14ff-15f5-422f-9dfe-8c6bfd9425b4 2024-09-18T15:11:22.837 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:11:22.838 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:11:22.838 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:11:22.838 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:11:22.838 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:11:22.842 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local _recovery_ops 2024-09-18T15:11:22.842 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:11:22.842 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:11:22.843 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:11:22.843 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:11:22.843 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:11:22.843 INFO:tasks.workunit.client.0.smithi046.stderr:226899: 2024-09-18T15:11:20.761+0000 7f487dd85540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:22.843 INFO:tasks.workunit.client.0.smithi046.stderr:226899: 2024-09-18T15:11:20.765+0000 7f487dd85540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:22.843 INFO:tasks.workunit.client.0.smithi046.stderr:226899: 2024-09-18T15:11:20.769+0000 7f487dd85540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:22.843 INFO:tasks.workunit.client.0.smithi046.stderr:226899: 2024-09-18T15:11:20.777+0000 7f487dd85540 -1 Falling back to public interface 2024-09-18T15:11:22.844 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:11:22.844 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:11:22.844 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:11:22.844 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:11:22.844 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:11:22.844 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:22.844 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:11:22.844 INFO:tasks.workunit.client.0.smithi046.stderr:226899: 0 2024-09-18T15:11:22.844 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:22.844 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:11:22.844 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:22.844 INFO:tasks.workunit.client.0.smithi046.stderr:226899: 2024-09-18T15:11:21.493+0000 7f487dd85540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:11:22.844 INFO:tasks.workunit.client.0.smithi046.stderr:226899: 2024-09-18T15:11:21.509+0000 7f487dd85540 -1 journal do_read_entry(1343488): bad header magic 2024-09-18T15:11:22.845 INFO:tasks.workunit.client.0.smithi046.stderr:226899: 2024-09-18T15:11:21.509+0000 7f487dd85540 -1 journal do_read_entry(1343488): bad header magic 2024-09-18T15:11:22.845 INFO:tasks.workunit.client.0.smithi046.stderr:226899: 2024-09-18T15:11:21.525+0000 7f487dd85540 -1 osd.1 85 log_to_monitors true 2024-09-18T15:11:22.845 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:22.845 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:22.845 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:11:22.845 INFO:tasks.workunit.client.0.smithi046.stderr:226899: 1 2024-09-18T15:11:22.845 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:22.845 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:11:22.845 INFO:tasks.workunit.client.0.smithi046.stderr:226899: osd.1 up in weight 1 up_from 90 up_thru 90 down_at 86 last_clean_interval [83,85) [v2:127.0.0.1:6810/227035,v1:127.0.0.1:6811/227035] [v2:127.0.0.1:6812/227035,v1:127.0.0.1:6813/227035] exists,up 115f2623-03c8-4519-b5cd-6dfb4555432f 2024-09-18T15:11:22.845 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:11:22.845 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:11:22.845 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:11:22.845 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:11:22.846 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:11:22.846 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:11:24.258 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: num_active_clean=-1 2024-09-18T15:11:24.258 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:11:24.258 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:11:24.258 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:11:24.258 INFO:tasks.workunit.client.0.smithi046.stderr:226896: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:11:24.258 INFO:tasks.workunit.client.0.smithi046.stderr:226896: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:11:24.258 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:11:24.258 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:11:24.258 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:11:24.259 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:11:24.259 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:11:24.259 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:11:24.259 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:11:24.259 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:11:24.259 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:11:24.259 INFO:tasks.workunit.client.0.smithi046.stderr:226896: 1 2024-09-18T15:11:24.259 INFO:tasks.workunit.client.0.smithi046.stderr:226896: 2 2024-09-18T15:11:24.259 INFO:tasks.workunit.client.0.smithi046.stderr:226896: 3' 2024-09-18T15:11:24.259 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:11:24.259 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:11:24.259 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:11:24.259 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836523 2024-09-18T15:11:24.260 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836523 2024-09-18T15:11:24.260 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836523' 2024-09-18T15:11:24.260 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:11:24.260 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:11:24.260 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=386547056642 2024-09-18T15:11:24.260 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 386547056642 2024-09-18T15:11:24.260 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836523 1-386547056642' 2024-09-18T15:11:24.260 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:11:24.260 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:11:24.260 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=317827579917 2024-09-18T15:11:24.260 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 317827579917 2024-09-18T15:11:24.260 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836523 1-386547056642 2-317827579917' 2024-09-18T15:11:24.260 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:11:24.260 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:11:24.261 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=395136991234 2024-09-18T15:11:24.268 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: fluwait_for_clean: local cur_active_clean 2024-09-18T15:11:24.268 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:11:24.268 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:11:24.268 INFO:tasks.workunit.client.0.smithi046.stderr:226899: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:11:24.268 INFO:tasks.workunit.client.0.smithi046.stderr:226899: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:11:24.268 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:11:24.268 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:11:24.268 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:11:24.268 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:11:24.268 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:11:24.268 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:11:24.268 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:11:24.268 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:11:24.269 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:11:24.269 INFO:tasks.workunit.client.0.smithi046.stderr:226899: 1 2024-09-18T15:11:24.269 INFO:tasks.workunit.client.0.smithi046.stderr:226899: 2 2024-09-18T15:11:24.269 INFO:tasks.workunit.client.0.smithi046.stderr:226899: 3' 2024-09-18T15:11:24.269 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:11:24.269 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:11:24.269 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:11:24.269 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836522 2024-09-18T15:11:24.269 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836522 2024-09-18T15:11:24.269 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836522' 2024-09-18T15:11:24.269 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:11:24.269 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:11:24.269 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=386547056643 2024-09-18T15:11:24.270 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 386547056643 2024-09-18T15:11:24.270 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836522 1-386547056643' 2024-09-18T15:11:24.270 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:11:24.270 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:11:24.270 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=317827579916 2024-09-18T15:11:24.270 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 317827579916 2024-09-18T15:11:24.270 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836522 1-386547056643 2-317827579916' 2024-09-18T15:11:24.270 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:11:24.270 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:11:24.270 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=395136991235 2024-09-18T15:11:24.270 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 395136991235 2024-09-18T15:11:26.245 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836522 1-386547056643 2-317827579916 3-395136991235' 2024-09-18T15:11:26.245 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:11:26.245 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836522 2024-09-18T15:11:26.245 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:11:26.245 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:11:26.246 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836522 2024-09-18T15:11:26.246 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:11:26.246 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836522 2024-09-18T15:11:26.246 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836522' 2024-09-18T15:11:26.246 INFO:tasks.workunit.client.0.smithi046.stderr:226899: waiting osd.0 seq 21474836522 2024-09-18T15:11:26.246 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:11:26.246 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836521 -lt 21474836522 2024-09-18T15:11:26.246 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:11:26.246 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:11:26.246 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:11:26.246 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836523 -lt 21474836522 2024-09-18T15:11:26.246 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:11:26.246 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-386547056643 2024-09-18T15:11:26.246 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:11:26.247 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:11:26.247 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-386547056643 2024-09-18T15:11:26.247 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:11:26.247 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=386547056643 2024-09-18T15:11:26.247 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 386547056643' 2024-09-18T15:11:26.247 INFO:tasks.workunit.client.0.smithi046.stderr:226899: waiting osd.1 seq 386547056643 2024-09-18T15:11:26.247 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:11:26.247 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 386547056643 -lt 386547056643 2024-09-18T15:11:26.247 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:11:26.247 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:11:26.247 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-317827579916 2024-09-18T15:11:26.248 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:11:26.248 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-317827579916 2024-09-18T15:11:26.248 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:11:26.248 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=317827579916 2024-09-18T15:11:26.248 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 317827579916' 2024-09-18T15:11:26.248 INFO:tasks.workunit.client.0.smithi046.stderr:226899: waiting osd.2 seq 317827579916 2024-09-18T15:11:26.249 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: csh_pg_stats: test -z 395136991234 2024-09-18T15:11:26.250 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836523 1-386547056642 2-317827579917 3-395136991234' 2024-09-18T15:11:26.250 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:11:26.250 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:11:26.250 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836523 2024-09-18T15:11:26.250 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:11:26.250 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836523 2024-09-18T15:11:26.250 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:11:26.250 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836523 2024-09-18T15:11:26.250 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2024-09-18T15:11:26.250 INFO:tasks.workunit.client.0.smithi046.stderr:226896: waiting osd.0 seq 21474836523 2024-09-18T15:11:26.250 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:11:26.250 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836521 -lt 21474836523 2024-09-18T15:11:26.250 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:11:26.250 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:11:26.251 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:11:26.251 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836523 -lt 21474836523 2024-09-18T15:11:26.251 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:11:26.251 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:11:26.251 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-386547056642 2024-09-18T15:11:26.251 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:11:26.251 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-386547056642 2024-09-18T15:11:26.251 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:11:26.251 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=386547056642 2024-09-18T15:11:26.251 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 386547056642' 2024-09-18T15:11:26.251 INFO:tasks.workunit.client.0.smithi046.stderr:226896: waiting osd.1 seq 386547056642 2024-09-18T15:11:26.251 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:11:26.251 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 386547056643 -lt 386547056642 2024-09-18T15:11:26.251 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:11:26.252 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-317827579917 2024-09-18T15:11:26.252 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:11:26.252 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:11:26.252 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-317827579917 2024-09-18T15:11:26.252 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:11:26.252 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=317827579917 2024-09-18T15:11:26.252 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 317827579917' 2024-09-18T15:11:27.980 INFO:tasks.workunit.client.0.smithi046.stderr:226896: waiting osd.2 seq 317827579917 2024-09-18T15:11:27.980 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:11:27.980 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 317827579917 -lt 317827579917 2024-09-18T15:11:27.980 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:11:27.980 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-395136991234 2024-09-18T15:11:27.980 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:11:27.980 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:11:27.981 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-395136991234 2024-09-18T15:11:27.981 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:11:27.981 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=395136991234 2024-09-18T15:11:27.981 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 395136991234' 2024-09-18T15:11:27.981 INFO:tasks.workunit.client.0.smithi046.stderr:226896: waiting osd.3 seq 395136991234 2024-09-18T15:11:27.981 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:11:27.981 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 395136991235 -lt 395136991234 2024-09-18T15:11:27.981 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:11:27.981 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:11:27.981 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:11:27.981 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:11:27.981 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:11:27.981 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:11:27.981 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:11:27.982 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:11:27.982 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:11:27.982 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:11:27.982 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:11:27.982 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:11:27.982 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:11:27.982 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:11:27.982 INFO:tasks.workunit.client.0.smithi046.stderr:226896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:11:27.982 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:11:27.982 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:11:27.982 INFO:tasks.workunit.client.0.smithi046.stderr:226896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:11:27.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:11:27.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:11:27.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 226895 2024-09-18T15:11:28.021 INFO:tasks.workunit.client.0.smithi046.stderr:eph osd last-stat-seq 2 2024-09-18T15:11:28.022 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 317827579917 -lt 317827579916 2024-09-18T15:11:28.022 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:11:28.022 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-395136991235 2024-09-18T15:11:28.022 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:11:28.022 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:11:28.022 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-395136991235 2024-09-18T15:11:28.022 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:11:28.022 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=395136991235 2024-09-18T15:11:28.022 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 395136991235' 2024-09-18T15:11:28.022 INFO:tasks.workunit.client.0.smithi046.stderr:226899: waiting osd.3 seq 395136991235 2024-09-18T15:11:28.023 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:11:28.023 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 395136991235 -lt 395136991235 2024-09-18T15:11:28.023 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:11:28.023 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:11:28.023 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:11:28.023 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:11:28.023 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:11:28.023 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:11:28.023 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:11:28.023 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:11:28.023 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:11:28.023 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:11:28.023 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:11:28.023 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:11:28.024 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:11:28.024 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:11:28.024 INFO:tasks.workunit.client.0.smithi046.stderr:226899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:11:28.024 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:11:28.024 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:11:28.024 INFO:tasks.workunit.client.0.smithi046.stderr:226899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:11:28.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:11:28.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: eval 'pids='\'''\''' 2024-09-18T15:11:28.025 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: pids= 2024-09-18T15:11:28.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: wait_background: return 0 2024-09-18T15:11:28.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: return_code=0 2024-09-18T15:11:28.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-18T15:11:28.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:214: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-18T15:11:28.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-18T15:11:28.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T15:11:28.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:11:28.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:11:28.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:11:28.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:11:28.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:11:28.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:11:28.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:11:28.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:11:28.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:11:28.204 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:11:28.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:11:28.206 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:11:28.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:11:28.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:11:28.210 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:11:28.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:11:28.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:11:28.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:11:28.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:11:28.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:11:28.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:11:28.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:11:28.252 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:11:28.252 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:11:28.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:11:28.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:11:28.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:11:28.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:11:28.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T15:11:28.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T15:11:28.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T15:11:28.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T15:11:28.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:11:28.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:11:28.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:11:28.256 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:11:28.256 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:11:28.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:11:28.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:11:28.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:11:28.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:11:28.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:11:28.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:11:28.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:11:28.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:11:28.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:11:28.266 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:11:28.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:11:28.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:11:28.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:11:28.269 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:11:28.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:11:28.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:11:28.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:11:28.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:11:28.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:11:28.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:11:28.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:11:28.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:11:28.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:11:28.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T15:11:28.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:11:28.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:11:28.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:11:28.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T15:11:28.283 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:11:28.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:11:28.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:11:28.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T15:11:28.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_corrupt_and_repair_jerasure_overwrites td/osd-scrub-repair 2024-09-18T15:11:28.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:824: TEST_corrupt_and_repair_jerasure_overwrites: '[' true = true ']' 2024-09-18T15:11:28.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: TEST_corrupt_and_repair_jerasure_overwrites: corrupt_and_repair_jerasure td/osd-scrub-repair true 2024-09-18T15:11:28.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-09-18T15:11:28.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local allow_overwrites=true 2024-09-18T15:11:28.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:801: corrupt_and_repair_jerasure: local poolname=ecpool 2024-09-18T15:11:28.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-09-18T15:11:28.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T15:11:28.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:11:28.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:11:28.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:11:28.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T15:11:28.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-18T15:11:28.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:11:28.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:11:28.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:11:28.700 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:11:28.700 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:11:28.700 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:11:28.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:11:28.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T15:11:28.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:11:28.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:11:28.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:11:28.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:11:28.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:11:28.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:11:28.983 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:11:28.983 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:11:28.983 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:11:28.983 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:11:28.984 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:11:28.984 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:11:28.984 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:11:28.984 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:11:28.984 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T15:11:29.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:11:29.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:11:29.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:11:29.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:11:29.076 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:11:29.076 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:11:29.076 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:11:29.076 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:11:29.078 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:11:29.078 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:11:29.078 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:11:29.078 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:11:29.079 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:11:29.079 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T15:11:29.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-09-18T15:11:29.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T15:11:29.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:11:29.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:11:29.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:11:29.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T15:11:29.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:11:30.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:11:30.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:11:30.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:11:30.031 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:11:30.031 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:11:30.031 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:11:30.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:11:30.032 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:11:30.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:11:30.065 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: seq 0 3 2024-09-18T15:11:30.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-18T15:11:30.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: corrupt_and_repair_jerasure: '[' true = true ']' 2024-09-18T15:11:30.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-09-18T15:11:30.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:11:30.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:11:30.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:11:30.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:11:30.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:11:30.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:11:30.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:11:30.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:11:30.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:11:30.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:11:30.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:11:30.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:11:30.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:11:30.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:11:30.068 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:11:30.068 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:11:30.069 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:11:30.071 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:11:30.071 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:11:30.071 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:11:30.071 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:11:30.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:11:30.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:11:30.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:11:30.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:11:30.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:11:30.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:11:30.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:11:30.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:11:30.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:11:30.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:11:30.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:11:30.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:11:30.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:11:30.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:11:30.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=01ed5ae6-9829-4c03-97e1-d175a9249e35 2024-09-18T15:11:30.074 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 01ed5ae6-9829-4c03-97e1-d175a9249e35 2024-09-18T15:11:30.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 01ed5ae6-9829-4c03-97e1-d175a9249e35' 2024-09-18T15:11:30.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:11:30.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCi7epmbJMsBRAABb+sm7Pm+TkU6/TTtCOzgQ== 2024-09-18T15:11:30.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCi7epmbJMsBRAABb+sm7Pm+TkU6/TTtCOzgQ=="}' 2024-09-18T15:11:30.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 01ed5ae6-9829-4c03-97e1-d175a9249e35 -i td/osd-scrub-repair/0/new.json 2024-09-18T15:11:30.380 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:11:30.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-18T15:11:30.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCi7epmbJMsBRAABb+sm7Pm+TkU6/TTtCOzgQ== --osd-uuid 01ed5ae6-9829-4c03-97e1-d175a9249e35 2024-09-18T15:11:30.420 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:11:30.409+0000 7f02d702a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:30.424 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:11:30.413+0000 7f02d702a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:30.426 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:11:30.417+0000 7f02d702a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:30.426 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:11:30.417+0000 7f02d702a540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:11:30.426 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:11:30.417+0000 7f02d702a540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:11:30.426 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:11:30.417+0000 7f02d702a540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:11:30.521 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:11:30.513+0000 7f02d702a540 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-18T15:11:36.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T15:11:36.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:11:36.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:11:36.676 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T15:11:36.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:11:37.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:11:37.056 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T15:11:37.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:11:37.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:11:37.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:11:37.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:11:37.088 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:11:37.077+0000 7fce08252540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:37.091 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:11:37.081+0000 7fce08252540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:37.093 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:11:37.085+0000 7fce08252540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:37.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:11:37.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:11:37.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:11:37.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:11:37.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:11:37.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:37.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:11:37.394 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:11:37.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:37.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:37.659 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:11:37.649+0000 7fce08252540 -1 Falling back to public interface 2024-09-18T15:11:37.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:38.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:38.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:38.858 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:11:38.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:11:38.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:38.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:39.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:40.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:40.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:40.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:11:40.186 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:11:40.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:40.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:40.335 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:11:40.325+0000 7fce08252540 -1 osd.0 0 log_to_monitors true 2024-09-18T15:11:40.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:41.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:41.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:41.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:11:41.509 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:11:41.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:41.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:41.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:42.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:42.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:42.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:11:42.847 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:11:42.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:42.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:43.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:44.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:44.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:44.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:11:44.284 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:11:44.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:44.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:44.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:45.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:45.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:45.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:11:45.599 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:11:45.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:45.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:45.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:46.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:46.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:46.923 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:11:46.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:11:46.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:46.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:47.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:48.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:48.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:48.309 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:11:48.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:11:48.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:48.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:48.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:49.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:49.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:49.624 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:11:49.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:11:49.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:49.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:49.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:50.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:50.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:50.939 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:11:50.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:11:50.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:50.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:51.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:52.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:52.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:52.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:11:52.335 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:11:52.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:52.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:52.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:53.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:53.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:53.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:11:53.654 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:11:53.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:53.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:54.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:55.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:55.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:55.061 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:11:55.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:11:55.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:55.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:55.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:56.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:56.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:56.694 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:11:56.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:11:56.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:56.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:57.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:58.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:58.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:58.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:11:58.393 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:11:58.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:58.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:58.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:59.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:59.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:59.785 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:11:59.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:11:59.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:59.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:12:00.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:01.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:01.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:01.153 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:12:01.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:12:01.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:01.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:12:01.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:02.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:02.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:02.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:12:02.956 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:12:02.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:02.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:12:03.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:04.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:04.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:04.276 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:12:04.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:12:04.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:04.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:12:04.717 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/229388,v1:127.0.0.1:6803/229388] [v2:127.0.0.1:6804/229388,v1:127.0.0.1:6805/229388] exists,up 01ed5ae6-9829-4c03-97e1-d175a9249e35 2024-09-18T15:12:04.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:12:04.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:12:04.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:12:04.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-18T15:12:04.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: corrupt_and_repair_jerasure: '[' true = true ']' 2024-09-18T15:12:04.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-09-18T15:12:04.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:12:04.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:12:04.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:12:04.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:12:04.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:12:04.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:12:04.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:12:04.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:12:04.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:12:04.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:12:04.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:12:04.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:12:04.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:12:04.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:12:04.720 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:12:04.720 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:12:04.720 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:12:04.721 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:12:04.721 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:12:04.722 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:12:04.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:12:04.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:12:04.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:12:04.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:12:04.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:12:04.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:12:04.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:12:04.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:12:04.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:12:04.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:12:04.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:12:04.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:12:04.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:12:04.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:12:04.723 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:12:04.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b59f73a0-2db8-4a9b-ae5e-0fcbadbde7ca 2024-09-18T15:12:04.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b59f73a0-2db8-4a9b-ae5e-0fcbadbde7ca' 2024-09-18T15:12:04.724 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 b59f73a0-2db8-4a9b-ae5e-0fcbadbde7ca 2024-09-18T15:12:04.725 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:12:04.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDE7epmewpGLBAA+1fsZo3So2dpkzxzWiTQHg== 2024-09-18T15:12:04.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDE7epmewpGLBAA+1fsZo3So2dpkzxzWiTQHg=="}' 2024-09-18T15:12:04.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b59f73a0-2db8-4a9b-ae5e-0fcbadbde7ca -i td/osd-scrub-repair/1/new.json 2024-09-18T15:12:05.180 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:12:05.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-18T15:12:05.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDE7epmewpGLBAA+1fsZo3So2dpkzxzWiTQHg== --osd-uuid b59f73a0-2db8-4a9b-ae5e-0fcbadbde7ca 2024-09-18T15:12:05.234 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:12:05.225+0000 7feec141c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:12:05.238 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:12:05.229+0000 7feec141c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:12:05.239 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:12:05.229+0000 7feec141c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:12:05.240 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:12:05.229+0000 7feec141c540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:12:05.240 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:12:05.229+0000 7feec141c540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:12:05.240 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:12:05.229+0000 7feec141c540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:12:05.320 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:12:05.309+0000 7feec141c540 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-18T15:12:12.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-18T15:12:12.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:12:12.161 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T15:12:12.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:12:12.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:12:12.545 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T15:12:12.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:12:12.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:12:12.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:12:12.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:12:12.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:12:12.578 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:12:12.569+0000 7fbddf981540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:12:12.580 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:12:12.569+0000 7fbddf981540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:12:12.581 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:12:12.573+0000 7fbddf981540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:12:12.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:12:12.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:12:12.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:12:12.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:12:12.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:12:12.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:12.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:12:12.842 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:12:12.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:12.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:12:13.111 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:12:13.101+0000 7fbddf981540 -1 Falling back to public interface 2024-09-18T15:12:13.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:14.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:14.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:14.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:12:14.145 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:12:14.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:14.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:12:14.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:15.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:15.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:15.455 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:12:15.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:12:15.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:15.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:12:15.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:15.985 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:12:15.977+0000 7fbddf981540 -1 osd.1 0 log_to_monitors true 2024-09-18T15:12:16.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:16.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:16.763 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:12:16.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:12:16.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:12:16.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:17.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:18.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:18.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:18.082 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:12:18.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:12:18.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:18.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:12:18.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:19.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:19.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:19.434 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:12:19.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:12:19.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:19.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:12:19.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:20.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:20.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:20.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:12:20.789 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:12:20.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:20.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:12:21.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:22.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:22.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:22.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:12:22.104 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:12:22.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:22.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:12:22.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:23.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:23.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:23.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:12:23.460 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:12:23.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:23.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:12:23.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:24.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:24.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:24.884 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:12:24.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:12:24.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:24.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:12:25.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:26.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:26.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:26.199 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:12:26.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:12:26.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:26.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:12:26.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:27.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:27.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:27.609 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:12:27.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:12:27.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:27.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:12:28.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:29.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:29.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:29.088 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:12:29.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:12:29.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:29.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:12:29.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:30.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:30.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:30.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:12:30.401 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:12:30.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:30.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:12:30.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:31.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:31.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:31.763 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:12:31.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:12:31.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:31.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:12:32.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:33.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:33.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:33.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:12:33.220 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:12:33.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:33.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:12:33.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:34.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:34.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:34.811 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:12:34.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:12:34.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:34.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:12:35.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:36.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:36.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:36.274 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:12:36.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:12:36.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:36.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:12:36.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:37.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:37.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:37.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:12:37.693 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:12:37.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:37.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:12:38.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:39.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:39.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:39.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:12:39.083 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:12:39.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:39.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:12:39.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:40.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:40.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:40.394 INFO:tasks.workunit.client.0.smithi046.stdout:20 2024-09-18T15:12:40.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-18T15:12:40.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:40.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:12:40.759 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/231193,v1:127.0.0.1:6811/231193] [v2:127.0.0.1:6812/231193,v1:127.0.0.1:6813/231193] exists,up b59f73a0-2db8-4a9b-ae5e-0fcbadbde7ca 2024-09-18T15:12:40.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:12:40.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:12:40.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:12:40.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-18T15:12:40.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: corrupt_and_repair_jerasure: '[' true = true ']' 2024-09-18T15:12:40.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-09-18T15:12:40.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:12:40.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:12:40.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T15:12:40.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:12:40.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:12:40.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:12:40.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:12:40.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:12:40.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:12:40.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T15:12:40.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T15:12:40.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:12:40.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:12:40.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:12:40.763 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:12:40.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:12:40.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:12:40.764 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:12:40.764 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:12:40.764 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:12:40.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:12:40.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:12:40.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:12:40.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:12:40.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:12:40.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:12:40.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:12:40.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:12:40.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:12:40.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:12:40.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:12:40.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:12:40.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:12:40.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-18T15:12:40.767 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:12:40.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=af350427-c715-418c-bd46-5204bfa0c41e 2024-09-18T15:12:40.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 af350427-c715-418c-bd46-5204bfa0c41e' 2024-09-18T15:12:40.767 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 af350427-c715-418c-bd46-5204bfa0c41e 2024-09-18T15:12:40.768 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:12:40.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDo7epmIMLULhAA/s/qDAbo85yPZEs10XeA2g== 2024-09-18T15:12:40.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDo7epmIMLULhAA/s/qDAbo85yPZEs10XeA2g=="}' 2024-09-18T15:12:40.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new af350427-c715-418c-bd46-5204bfa0c41e -i td/osd-scrub-repair/2/new.json 2024-09-18T15:12:41.179 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:12:41.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-18T15:12:41.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDo7epmIMLULhAA/s/qDAbo85yPZEs10XeA2g== --osd-uuid af350427-c715-418c-bd46-5204bfa0c41e 2024-09-18T15:12:41.229 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:12:41.221+0000 7fc39d67c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:12:41.237 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:12:41.229+0000 7fc39d67c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:12:41.239 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:12:41.229+0000 7fc39d67c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:12:41.240 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:12:41.229+0000 7fc39d67c540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T15:12:41.240 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:12:41.229+0000 7fc39d67c540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T15:12:41.240 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:12:41.229+0000 7fc39d67c540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T15:12:41.298 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:12:41.289+0000 7fc39d67c540 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-18T15:12:47.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-18T15:12:47.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:12:47.872 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T15:12:47.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T15:12:47.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:12:48.264 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T15:12:48.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T15:12:48.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:12:48.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:12:48.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:12:48.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:12:48.295 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:12:48.284+0000 7f7c13f51540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:12:48.309 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:12:48.300+0000 7f7c13f51540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:12:48.310 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:12:48.300+0000 7f7c13f51540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:12:48.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T15:12:48.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:12:48.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:12:48.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:12:48.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:12:48.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:48.558 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:12:48.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:12:48.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:48.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:12:48.864 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:12:48.852+0000 7f7c13f51540 -1 Falling back to public interface 2024-09-18T15:12:48.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:49.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:49.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:49.905 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:12:49.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:12:49.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:49.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:12:50.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:51.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:51.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:51.250 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:12:51.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:12:51.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:51.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:12:51.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:51.625 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:12:51.616+0000 7f7c13f51540 -1 osd.2 0 log_to_monitors true 2024-09-18T15:12:52.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:52.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:52.575 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:12:52.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:12:52.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:52.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:12:52.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:53.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:53.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:53.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:12:53.886 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:12:53.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:53.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:12:54.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:55.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:55.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:55.198 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:12:55.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:12:55.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:55.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:12:55.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:56.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:56.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:56.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:12:56.523 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:12:56.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:56.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:12:56.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:57.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:57.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:57.839 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:12:57.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:12:57.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:57.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:12:58.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:59.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:59.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:59.195 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:12:59.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:12:59.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:59.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:12:59.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:00.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:00.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:00.522 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:13:00.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:13:00.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:00.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:13:00.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:01.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:01.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:01.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:13:01.837 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:13:01.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:01.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:13:02.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:03.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:03.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:03.212 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:13:03.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:13:03.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:03.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:13:03.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:04.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:04.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:04.658 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:13:04.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:13:04.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:04.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:13:05.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:06.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:06.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:06.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:13:06.203 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:13:06.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:06.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:13:06.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:07.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:07.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:07.575 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:13:07.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:13:07.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:07.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:13:08.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:09.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:09.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:09.014 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:13:09.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:13:09.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:09.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:13:09.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:10.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:10.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:10.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:13:10.504 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:13:10.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:10.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:13:10.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:11.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:11.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:11.897 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:13:11.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:13:11.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:11.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:13:12.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:13.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:13.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:13.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:13:13.490 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:13:13.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:13.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:13:14.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:15.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:15.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:15.070 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:13:15.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:13:15.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:15.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:13:15.385 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/233022,v1:127.0.0.1:6819/233022] [v2:127.0.0.1:6820/233022,v1:127.0.0.1:6821/233022] exists,up af350427-c715-418c-bd46-5204bfa0c41e 2024-09-18T15:13:15.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:13:15.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:13:15.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:13:15.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-18T15:13:15.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: corrupt_and_repair_jerasure: '[' true = true ']' 2024-09-18T15:13:15.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-09-18T15:13:15.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:13:15.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:13:15.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T15:13:15.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:13:15.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:13:15.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:13:15.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:13:15.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:13:15.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:13:15.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-18T15:13:15.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-18T15:13:15.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:13:15.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:13:15.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:13:15.389 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:13:15.389 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:13:15.389 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:13:15.391 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:13:15.391 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:13:15.391 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:13:15.391 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:13:15.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:13:15.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:13:15.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:13:15.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:13:15.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:13:15.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:13:15.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:13:15.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:13:15.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:13:15.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:13:15.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:13:15.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:13:15.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-09-18T15:13:15.392 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:13:15.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=db129174-c3ee-42d9-b16d-41a610ac05c9 2024-09-18T15:13:15.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 db129174-c3ee-42d9-b16d-41a610ac05c9' 2024-09-18T15:13:15.394 INFO:tasks.workunit.client.0.smithi046.stdout:add osd3 db129174-c3ee-42d9-b16d-41a610ac05c9 2024-09-18T15:13:15.394 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:13:15.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAL7upmmw5bGBAARI/m8Hlpwtq+P+0xVEypqQ== 2024-09-18T15:13:15.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAL7upmmw5bGBAARI/m8Hlpwtq+P+0xVEypqQ=="}' 2024-09-18T15:13:15.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new db129174-c3ee-42d9-b16d-41a610ac05c9 -i td/osd-scrub-repair/3/new.json 2024-09-18T15:13:16.012 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:13:16.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-09-18T15:13:16.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAL7upmmw5bGBAARI/m8Hlpwtq+P+0xVEypqQ== --osd-uuid db129174-c3ee-42d9-b16d-41a610ac05c9 2024-09-18T15:13:16.069 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:13:16.060+0000 7f06a7aca540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:13:16.072 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:13:16.064+0000 7f06a7aca540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:13:16.075 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:13:16.064+0000 7f06a7aca540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:13:16.076 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:13:16.064+0000 7f06a7aca540 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-09-18T15:13:16.076 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:13:16.064+0000 7f06a7aca540 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-09-18T15:13:16.076 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:13:16.064+0000 7f06a7aca540 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-09-18T15:13:16.144 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:13:16.136+0000 7f06a7aca540 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-09-18T15:13:22.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-18T15:13:22.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:13:22.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T15:13:22.725 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd3 key to auth repository 2024-09-18T15:13:22.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:13:23.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T15:13:23.108 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.3 2024-09-18T15:13:23.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:13:23.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:13:23.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:13:23.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:13:23.141 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:13:23.132+0000 7f347d11f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:13:23.143 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:13:23.132+0000 7f347d11f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:13:23.144 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:13:23.136+0000 7f347d11f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:13:23.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T15:13:23.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:13:23.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:13:23.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:13:23.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:13:23.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:23.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:13:23.466 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:13:23.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:23.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:23.700 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:13:23.688+0000 7f347d11f540 -1 Falling back to public interface 2024-09-18T15:13:23.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:24.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:24.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:24.818 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:13:24.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:13:24.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:24.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:25.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:26.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:26.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:26.141 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:13:26.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:13:26.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:26.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:26.282 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:13:26.272+0000 7f347d11f540 -1 osd.3 0 log_to_monitors true 2024-09-18T15:13:26.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:27.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:27.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:27.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:13:27.553 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:13:27.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:27.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:27.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:28.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:28.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:28.867 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:13:28.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:13:28.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:28.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:29.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:30.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:30.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:30.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:13:30.181 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:13:30.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:30.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:30.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:31.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:31.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:31.665 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:13:31.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:13:31.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:31.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:31.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:32.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:32.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:32.989 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:13:32.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:13:32.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:32.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:33.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:34.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:34.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:34.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:13:34.581 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:13:34.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:34.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:34.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:35.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:35.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:35.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:13:35.899 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:13:35.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:35.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:36.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:37.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:37.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:37.286 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:13:37.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:13:37.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:37.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:37.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:38.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:38.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:38.604 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:13:38.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:13:38.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:38.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:38.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:39.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:39.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:39.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:13:39.919 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:13:39.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:39.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:40.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:41.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:41.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:41.247 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:13:41.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:13:41.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:41.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:41.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:42.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:42.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:42.852 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:13:42.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:13:42.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:42.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:43.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:44.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:44.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:44.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:13:44.255 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:13:44.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:44.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:44.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:45.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:45.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:45.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:13:45.583 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:13:45.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:45.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:46.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:47.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:47.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:47.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:13:47.338 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:13:47.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:47.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:47.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:48.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:48.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:48.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:13:48.862 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:13:48.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:48.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:49.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:50.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:50.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:50.586 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:13:50.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:13:50.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:50.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:50.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:51.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:51.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:51.985 INFO:tasks.workunit.client.0.smithi046.stdout:20 2024-09-18T15:13:51.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-18T15:13:51.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:51.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:52.291 INFO:tasks.workunit.client.0.smithi046.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/234826,v1:127.0.0.1:6827/234826] [v2:127.0.0.1:6828/234826,v1:127.0.0.1:6829/234826] exists,up db129174-c3ee-42d9-b16d-41a610ac05c9 2024-09-18T15:13:52.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:13:52.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:13:52.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:13:52.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:812: corrupt_and_repair_jerasure: create_rbd_pool 2024-09-18T15:13:52.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T15:13:52.697 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' does not exist 2024-09-18T15:13:52.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T15:13:52.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T15:13:53.212 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' created 2024-09-18T15:13:53.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:13:54.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T15:13:56.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:813: corrupt_and_repair_jerasure: wait_for_clean 2024-09-18T15:13:56.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:13:56.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:13:56.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:13:56.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:13:56.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:13:56.604 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:13:56.604 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:13:56.605 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:13:56.605 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:13:56.605 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:13:56.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:13:56.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:13:56.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:13:56.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:13:56.839 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:13:57.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:13:57.154 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:13:57.155 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:13:57.155 INFO:tasks.workunit.client.0.smithi046.stderr:3' 2024-09-18T15:13:57.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:13:57.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:13:57.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:13:57.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836511 2024-09-18T15:13:57.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836511 2024-09-18T15:13:57.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836511' 2024-09-18T15:13:57.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:13:57.330 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:13:57.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672981 2024-09-18T15:13:57.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672981 2024-09-18T15:13:57.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836511 1-42949672981' 2024-09-18T15:13:57.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:13:57.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:13:57.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509455 2024-09-18T15:13:57.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509455 2024-09-18T15:13:57.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836511 1-42949672981 2-64424509455' 2024-09-18T15:13:57.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:13:57.611 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:13:57.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345927 2024-09-18T15:13:57.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345927 2024-09-18T15:13:57.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836511 1-42949672981 2-64424509455 3-85899345927' 2024-09-18T15:13:57.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:13:57.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836511 2024-09-18T15:13:57.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:13:57.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:13:57.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836511 2024-09-18T15:13:57.753 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:13:57.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836511 2024-09-18T15:13:57.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-09-18T15:13:57.755 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836511 2024-09-18T15:13:57.756 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:13:58.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836511 -lt 21474836511 2024-09-18T15:13:58.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:13:58.071 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672981 2024-09-18T15:13:58.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:13:58.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:13:58.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672981 2024-09-18T15:13:58.076 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:13:58.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672981 2024-09-18T15:13:58.078 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672981 2024-09-18T15:13:58.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-09-18T15:13:58.079 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:13:58.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672980 -lt 42949672981 2024-09-18T15:13:58.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:13:59.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:13:59.432 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:13:59.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672981 -lt 42949672981 2024-09-18T15:13:59.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:13:59.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509455 2024-09-18T15:13:59.742 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:13:59.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:13:59.745 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509455 2024-09-18T15:13:59.745 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:13:59.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509455 2024-09-18T15:13:59.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-18T15:13:59.747 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509455 2024-09-18T15:13:59.748 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:14:00.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-18T15:14:00.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:14:00.092 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345927 2024-09-18T15:14:00.092 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:14:00.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:14:00.095 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345927 2024-09-18T15:14:00.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:14:00.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345927 2024-09-18T15:14:00.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345927' 2024-09-18T15:14:00.098 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.3 seq 85899345927 2024-09-18T15:14:00.099 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:14:00.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345928 -lt 85899345927 2024-09-18T15:14:00.439 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:14:00.439 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:14:00.439 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:14:00.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T15:14:00.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:14:00.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:14:00.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:14:00.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:14:00.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:14:00.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:14:00.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:14:01.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T15:14:01.237 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:14:01.237 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:14:01.238 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:14:01.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T15:14:01.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:14:01.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:14:01.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:815: corrupt_and_repair_jerasure: create_ec_pool ecpool true k=2 m=2 2024-09-18T15:14:01.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: create_ec_pool: local pool_name=ecpool 2024-09-18T15:14:01.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: create_ec_pool: shift 2024-09-18T15:14:01.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: create_ec_pool: local allow_overwrites=true 2024-09-18T15:14:01.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: create_ec_pool: shift 2024-09-18T15:14:01.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-09-18T15:14:02.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-18T15:14:02.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-18T15:14:03.020 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'ecpool' created 2024-09-18T15:14:03.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:14:04.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: create_ec_pool: '[' true = true ']' 2024-09-18T15:14:04.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-18T15:14:04.837 INFO:tasks.workunit.client.0.smithi046.stderr:set pool 2 allow_ec_overwrites to true 2024-09-18T15:14:04.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: create_ec_pool: wait_for_clean 2024-09-18T15:14:04.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:14:04.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:14:04.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:14:04.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:14:04.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:14:04.850 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:14:04.850 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:14:04.851 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:14:04.851 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:14:04.851 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:14:05.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:14:05.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:14:05.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:14:05.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:14:05.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:14:05.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:14:05.390 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:14:05.390 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:14:05.390 INFO:tasks.workunit.client.0.smithi046.stderr:3' 2024-09-18T15:14:05.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:14:05.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:14:05.391 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:14:05.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836514 2024-09-18T15:14:05.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836514 2024-09-18T15:14:05.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836514' 2024-09-18T15:14:05.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:14:05.578 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:14:05.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672983 2024-09-18T15:14:05.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672983 2024-09-18T15:14:05.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836514 1-42949672983' 2024-09-18T15:14:05.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:14:05.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:14:05.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509457 2024-09-18T15:14:05.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509457 2024-09-18T15:14:05.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836514 1-42949672983 2-64424509457' 2024-09-18T15:14:05.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:14:05.854 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:14:05.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345930 2024-09-18T15:14:05.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345930 2024-09-18T15:14:05.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836514 1-42949672983 2-64424509457 3-85899345930' 2024-09-18T15:14:05.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:14:05.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836514 2024-09-18T15:14:05.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:14:05.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:14:05.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836514 2024-09-18T15:14:05.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:14:05.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836514 2024-09-18T15:14:05.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-09-18T15:14:05.998 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836514 2024-09-18T15:14:05.999 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:14:06.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836513 -lt 21474836514 2024-09-18T15:14:06.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:14:07.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:14:07.315 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:14:07.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836514 -lt 21474836514 2024-09-18T15:14:07.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:14:07.633 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672983 2024-09-18T15:14:07.633 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:14:07.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:14:07.637 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672983 2024-09-18T15:14:07.637 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:14:07.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672983 2024-09-18T15:14:07.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-09-18T15:14:07.640 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672983 2024-09-18T15:14:07.640 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:14:07.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672984 -lt 42949672983 2024-09-18T15:14:07.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:14:07.943 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509457 2024-09-18T15:14:07.943 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:14:07.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:14:07.946 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509457 2024-09-18T15:14:07.947 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:14:07.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509457 2024-09-18T15:14:07.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-09-18T15:14:07.949 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509457 2024-09-18T15:14:07.949 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:14:08.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509458 -lt 64424509457 2024-09-18T15:14:08.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:14:08.322 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345930 2024-09-18T15:14:08.322 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:14:08.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:14:08.324 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345930 2024-09-18T15:14:08.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:14:08.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345930 2024-09-18T15:14:08.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345930' 2024-09-18T15:14:08.327 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.3 seq 85899345930 2024-09-18T15:14:08.328 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:14:08.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345930 -lt 85899345930 2024-09-18T15:14:08.670 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:14:08.671 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:14:08.671 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:14:09.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:14:09.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:14:09.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:14:09.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:14:09.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:14:09.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:14:09.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:14:09.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:14:09.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:14:09.416 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:14:09.417 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:14:09.417 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:14:09.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:14:09.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:14:09.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:14:09.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: create_ec_pool: return 0 2024-09-18T15:14:09.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:816: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-09-18T15:14:09.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-18T15:14:09.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-09-18T15:14:09.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:250: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-18T15:14:09.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:14:09.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2024-09-18T15:14:09.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=SOMETHING 2024-09-18T15:14:09.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:14:09.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:14:09.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:14:10.332 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:14:10.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:14:10.934 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:14:10.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:14:10.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:14:10.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-18T15:14:11.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:252: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-09-18T15:14:11.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-09-18T15:14:11.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T15:14:11.192 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:14:11.192 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:14:11.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:252: corrupt_and_repair_erasure_coded: local primary=3 2024-09-18T15:14:11.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: osds=($(get_osds $poolname SOMETHING | sed -e "s/$primary//")) 2024-09-18T15:14:11.546 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-09-18T15:14:11.546 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-09-18T15:14:11.546 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-18T15:14:11.546 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-09-18T15:14:11.546 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:14:11.547 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:14:11.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:14:11.885 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:14:11.885 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:14:11.885 INFO:tasks.workunit.client.0.smithi046.stderr:0' 2024-09-18T15:14:11.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 0 2024-09-18T15:14:11.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local -a osds 2024-09-18T15:14:11.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-09-18T15:14:11.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-09-18T15:14:11.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-09-18T15:14:11.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:225: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-18T15:14:11.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local poolname=ecpool 2024-09-18T15:14:11.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local osd=3 2024-09-18T15:14:11.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:14:11.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:14:11.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:14:11.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-18T15:14:11.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:14:11.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:14:11.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:14:11.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:14:11.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-18T15:14:11.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:14:11.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-18T15:14:11.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:14:11.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:14:11.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:14:11.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:14:11.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:14:12.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:14:12.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:14:12.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:14:12.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:14:12.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-18T15:14:12.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:14:12.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:14:12.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-18T15:14:14.539 INFO:tasks.workunit.client.0.smithi046.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-09-18T15:14:15.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-18T15:14:15.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:14:15.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:14:15.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T15:14:15.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:14:15.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:14:15.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:14:15.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:14:15.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:14:15.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:14:15.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-18T15:14:15.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-18T15:14:15.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:14:15.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:14:15.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:14:15.077 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:14:15.077 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:14:15.077 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:14:15.079 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:14:15.079 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:14:15.079 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:14:15.079 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:14:15.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:14:15.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:14:15.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:14:15.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:14:15.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:14:15.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:14:15.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:14:15.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:14:15.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:14:15.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:14:15.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-18T15:14:15.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T15:14:15.081 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.3 2024-09-18T15:14:15.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:14:15.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-18T15:14:15.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T15:14:15.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:14:15.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:14:15.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:14:15.111 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:14:15.103+0000 7fbe9c36a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:14:15.115 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:14:15.107+0000 7fbe9c36a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:14:15.124 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:14:15.111+0000 7fbe9c36a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:14:15.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T15:14:15.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:14:15.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:14:15.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:14:15.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:14:15.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:14:15.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:14:15.384 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:14:15.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:14:15.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:14:15.411 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:14:15.403+0000 7fbe9c36a540 -1 Falling back to public interface 2024-09-18T15:14:15.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:14:16.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:14:16.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:14:16.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:14:16.796 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:14:16.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:14:16.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:14:17.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:14:18.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:14:18.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:14:18.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:14:18.111 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:14:18.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:14:18.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:14:18.217 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:14:18.211+0000 7fbe9c36a540 -1 osd.3 38 log_to_monitors true 2024-09-18T15:14:18.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:14:19.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:14:19.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:14:19.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:14:19.423 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:14:19.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:14:19.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:14:20.058 INFO:tasks.workunit.client.0.smithi046.stderr:osd.3 up in weight 1 up_from 43 up_thru 43 down_at 39 last_clean_interval [20,38) [v2:127.0.0.1:6826/237347,v1:127.0.0.1:6827/237347] [v2:127.0.0.1:6828/237347,v1:127.0.0.1:6829/237347] exists,up db129174-c3ee-42d9-b16d-41a610ac05c9 2024-09-18T15:14:20.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:14:20.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:14:20.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:14:20.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:14:20.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:14:20.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:14:20.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:14:20.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:14:20.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:14:20.060 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:14:20.060 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:14:20.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:14:20.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:14:20.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:14:20.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:14:20.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:14:20.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:14:20.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:14:20.277 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:14:20.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:14:20.587 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:14:20.587 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:14:20.587 INFO:tasks.workunit.client.0.smithi046.stderr:3' 2024-09-18T15:14:20.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:14:20.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:14:20.588 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:14:20.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836518 2024-09-18T15:14:20.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836518 2024-09-18T15:14:20.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836518' 2024-09-18T15:14:20.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:14:20.769 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:14:20.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672987 2024-09-18T15:14:20.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672987 2024-09-18T15:14:20.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836518 1-42949672987' 2024-09-18T15:14:20.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:14:20.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:14:21.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509461 2024-09-18T15:14:21.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509461 2024-09-18T15:14:21.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836518 1-42949672987 2-64424509461' 2024-09-18T15:14:21.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:14:21.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:14:21.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=184683593730 2024-09-18T15:14:21.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 184683593730 2024-09-18T15:14:21.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836518 1-42949672987 2-64424509461 3-184683593730' 2024-09-18T15:14:21.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:14:21.198 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836518 2024-09-18T15:14:21.198 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:14:21.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:14:21.200 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836518 2024-09-18T15:14:21.200 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:14:21.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836518 2024-09-18T15:14:21.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2024-09-18T15:14:21.202 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836518 2024-09-18T15:14:21.202 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:14:21.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836516 -lt 21474836518 2024-09-18T15:14:21.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:14:22.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:14:22.517 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:14:22.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836518 -lt 21474836518 2024-09-18T15:14:22.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:14:22.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672987 2024-09-18T15:14:22.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:14:22.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:14:22.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672987 2024-09-18T15:14:22.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:14:22.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672987 2024-09-18T15:14:22.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-09-18T15:14:22.837 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 42949672987 2024-09-18T15:14:22.837 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:14:23.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672987 -lt 42949672987 2024-09-18T15:14:23.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:14:23.180 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509461 2024-09-18T15:14:23.180 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:14:23.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:14:23.184 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509461 2024-09-18T15:14:23.184 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:14:23.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509461 2024-09-18T15:14:23.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-09-18T15:14:23.185 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509461 2024-09-18T15:14:23.186 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:14:23.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509462 -lt 64424509461 2024-09-18T15:14:23.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:14:23.525 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-184683593730 2024-09-18T15:14:23.525 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:14:23.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:14:23.527 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-184683593730 2024-09-18T15:14:23.528 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:14:23.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=184683593730 2024-09-18T15:14:23.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 184683593730' 2024-09-18T15:14:23.533 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.3 seq 184683593730 2024-09-18T15:14:23.534 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:14:23.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 184683593730 -lt 184683593730 2024-09-18T15:14:23.858 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:14:23.859 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:14:23.859 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:14:24.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:14:24.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:14:24.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:14:24.274 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:14:24.274 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:14:24.274 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:14:24.274 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:14:24.274 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:14:24.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:14:24.576 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:14:24.577 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:14:24.577 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:14:24.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:14:24.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:14:24.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:14:24.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-18T15:14:24.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-09-18T15:14:24.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T15:14:24.952 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:14:24.952 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:14:25.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: local pg=2.0 2024-09-18T15:14:25.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: repair 2.0 2024-09-18T15:14:25.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-09-18T15:14:25.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-09-18T15:14:25.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:14:25.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:14:25.276 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:14:25.276 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:14:25.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T15:14:02.863344+0000 2024-09-18T15:14:25.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-09-18T15:14:25.912 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-18T15:14:25.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-09-18T15:14:02.863344+0000 2024-09-18T15:14:25.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=2.0 2024-09-18T15:14:25.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:14:02.863344+0000 2024-09-18T15:14:25.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:14:25.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:14:25.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:14:25.922 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:14:25.922 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:14:25.922 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:14:25.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:14:25.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:14:26.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:14:02.863344+0000 '>' 2024-09-18T15:14:02.863344+0000 2024-09-18T15:14:26.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:14:27.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:14:27.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:14:27.233 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:14:27.233 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:14:27.233 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:14:27.233 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:14:27.233 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:14:27.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:14:02.863344+0000 '>' 2024-09-18T15:14:02.863344+0000 2024-09-18T15:14:27.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:14:28.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:14:28.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:14:28.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:14:28.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:14:28.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:14:28.568 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:14:28.568 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:14:28.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:14:02.863344+0000 '>' 2024-09-18T15:14:02.863344+0000 2024-09-18T15:14:28.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:14:29.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:14:29.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:14:29.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:14:29.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:14:29.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:14:29.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:14:29.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:14:30.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:14:26.234869+0000 '>' 2024-09-18T15:14:02.863344+0000 2024-09-18T15:14:30.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:14:30.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:241: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:14:30.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:14:30.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:14:30.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-18T15:14:30.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:14:30.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:14:30.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:14:30.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:14:30.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-18T15:14:30.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:14:30.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-18T15:14:30.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:14:30.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:14:30.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:14:30.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:14:30.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:14:31.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:14:31.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:14:31.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:14:31.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:14:31.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-18T15:14:31.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:14:31.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:14:31.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-18T15:14:32.895 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-18T15:14:32.895 INFO:tasks.workunit.client.0.smithi046.stdout:_ 2024-09-18T15:14:32.895 INFO:tasks.workunit.client.0.smithi046.stdout:hinfo_key 2024-09-18T15:14:32.895 INFO:tasks.workunit.client.0.smithi046.stdout:snapset 2024-09-18T15:14:33.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-18T15:14:33.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:14:33.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:14:33.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T15:14:33.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:14:33.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:14:33.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:14:33.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:14:33.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:14:33.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:14:33.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-18T15:14:33.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-18T15:14:33.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:14:33.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:14:33.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:14:33.433 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:14:33.433 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:14:33.433 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:14:33.434 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:14:33.435 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:14:33.435 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:14:33.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:14:33.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:14:33.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:14:33.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:14:33.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:14:33.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:14:33.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:14:33.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:14:33.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:14:33.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:14:33.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:14:33.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-18T15:14:33.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T15:14:33.436 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.3 2024-09-18T15:14:33.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:14:33.437 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-18T15:14:33.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T15:14:33.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:14:33.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:14:33.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:14:33.466 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:14:33.459+0000 7ffa91a16540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:14:33.469 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:14:33.463+0000 7ffa91a16540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:14:33.471 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:14:33.463+0000 7ffa91a16540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:14:33.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T15:14:33.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:14:33.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:14:33.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:14:33.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:14:33.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:14:33.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:14:33.780 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:14:33.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:14:33.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:14:34.023 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:14:34.015+0000 7ffa91a16540 -1 Falling back to public interface 2024-09-18T15:14:34.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:14:35.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:14:35.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:14:35.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:14:35.196 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:14:35.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:14:35.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:14:35.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:14:36.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:14:36.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:14:36.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:14:36.520 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:14:36.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:14:36.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:14:36.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:14:36.925 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:14:36.919+0000 7ffa91a16540 -1 osd.3 45 log_to_monitors true 2024-09-18T15:14:37.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:14:37.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:14:37.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:14:37.838 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:14:37.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:14:37.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:14:38.494 INFO:tasks.workunit.client.0.smithi046.stderr:osd.3 up in weight 1 up_from 50 up_thru 50 down_at 46 last_clean_interval [43,45) [v2:127.0.0.1:6826/238930,v1:127.0.0.1:6827/238930] [v2:127.0.0.1:6828/238930,v1:127.0.0.1:6829/238930] exists,up db129174-c3ee-42d9-b16d-41a610ac05c9 2024-09-18T15:14:38.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:14:38.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:14:38.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:14:38.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:14:38.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:14:38.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:14:38.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:14:38.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:14:38.496 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:14:38.497 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:14:38.497 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:14:38.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:14:38.498 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:14:38.498 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:14:38.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:14:38.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:14:38.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:14:38.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:14:38.720 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:14:39.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:14:39.034 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:14:39.034 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:14:39.034 INFO:tasks.workunit.client.0.smithi046.stderr:3' 2024-09-18T15:14:39.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:14:39.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:14:39.035 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:14:39.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836522 2024-09-18T15:14:39.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836522 2024-09-18T15:14:39.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836522' 2024-09-18T15:14:39.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:14:39.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:14:39.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672992 2024-09-18T15:14:39.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672992 2024-09-18T15:14:39.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836522 1-42949672992' 2024-09-18T15:14:39.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:14:39.350 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:14:39.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509466 2024-09-18T15:14:39.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509466 2024-09-18T15:14:39.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836522 1-42949672992 2-64424509466' 2024-09-18T15:14:39.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:14:39.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:14:39.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=214748364802 2024-09-18T15:14:39.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 214748364802 2024-09-18T15:14:39.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836522 1-42949672992 2-64424509466 3-214748364802' 2024-09-18T15:14:39.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:14:39.690 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836522 2024-09-18T15:14:39.690 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:14:39.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:14:39.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836522 2024-09-18T15:14:39.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:14:39.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836522 2024-09-18T15:14:39.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836522' 2024-09-18T15:14:39.695 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836522 2024-09-18T15:14:39.696 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:14:40.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836522 -lt 21474836522 2024-09-18T15:14:40.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:14:40.010 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672992 2024-09-18T15:14:40.010 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:14:40.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:14:40.013 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672992 2024-09-18T15:14:40.013 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:14:40.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672992 2024-09-18T15:14:40.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672992' 2024-09-18T15:14:40.014 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 42949672992 2024-09-18T15:14:40.015 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:14:40.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672992 -lt 42949672992 2024-09-18T15:14:40.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:14:40.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509466 2024-09-18T15:14:40.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:14:40.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:14:40.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509466 2024-09-18T15:14:40.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:14:40.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509466 2024-09-18T15:14:40.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2024-09-18T15:14:40.343 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509466 2024-09-18T15:14:40.343 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:14:40.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509465 -lt 64424509466 2024-09-18T15:14:40.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:14:41.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:14:41.669 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:14:41.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509467 -lt 64424509466 2024-09-18T15:14:41.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:14:41.990 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-214748364802 2024-09-18T15:14:41.991 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:14:41.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:14:41.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-214748364802 2024-09-18T15:14:41.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:14:41.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=214748364802 2024-09-18T15:14:41.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 214748364802' 2024-09-18T15:14:41.996 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.3 seq 214748364802 2024-09-18T15:14:41.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:14:42.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 214748364802 -lt 214748364802 2024-09-18T15:14:42.338 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:14:42.338 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:14:42.339 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:14:42.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:14:42.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:14:42.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:14:42.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:14:42.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:14:42.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:14:42.742 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:14:42.742 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:14:43.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:14:43.124 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:14:43.125 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:14:43.125 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:14:43.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:14:43.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:14:43.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:14:43.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-18T15:14:43.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-18T15:14:43.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:260: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-09-18T15:14:43.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:225: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-18T15:14:43.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local poolname=ecpool 2024-09-18T15:14:43.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local osd=1 2024-09-18T15:14:43.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:14:43.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:14:43.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:14:43.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:14:43.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:14:43.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:14:43.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:14:43.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:14:43.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:14:43.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:14:43.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:14:43.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:14:43.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:14:43.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:14:43.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:14:43.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:14:44.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:14:44.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:14:44.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:14:44.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:14:44.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:14:44.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:14:44.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:14:44.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-18T15:14:46.746 INFO:tasks.workunit.client.0.smithi046.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-18T15:14:47.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:14:47.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:14:47.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:14:47.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:14:47.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:14:47.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:14:47.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:14:47.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:14:47.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:14:47.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:14:47.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:14:47.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:14:47.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:14:47.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:14:47.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:14:47.288 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:14:47.288 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:14:47.288 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:14:47.289 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:14:47.289 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:14:47.289 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:14:47.289 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:14:47.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:14:47.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:14:47.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:14:47.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:14:47.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:14:47.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:14:47.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:14:47.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:14:47.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:14:47.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:14:47.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:14:47.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:14:47.290 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:14:47.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:14:47.290 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:14:47.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:14:47.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:14:47.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:14:47.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:14:47.320 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:14:47.311+0000 7f52a7989540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:14:47.324 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:14:47.315+0000 7f52a7989540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:14:47.325 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:14:47.319+0000 7f52a7989540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:14:47.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:14:47.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:14:47.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:14:47.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:14:47.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:14:47.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:14:47.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:14:47.605 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:14:47.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:14:47.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:14:47.875 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:14:47.867+0000 7f52a7989540 -1 Falling back to public interface 2024-09-18T15:14:47.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:14:48.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:14:48.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:14:48.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:14:48.955 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:14:48.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:14:48.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:14:49.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:14:50.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:14:50.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:14:50.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:14:50.264 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:14:50.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:14:50.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:14:50.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:14:50.861 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:14:50.855+0000 7f52a7989540 -1 osd.1 52 log_to_monitors true 2024-09-18T15:14:51.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:14:51.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:14:51.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:14:51.582 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:14:51.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:14:51.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:14:52.130 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 53 last_clean_interval [10,52) [v2:127.0.0.1:6810/240358,v1:127.0.0.1:6811/240358] [v2:127.0.0.1:6812/240358,v1:127.0.0.1:6813/240358] exists,up b59f73a0-2db8-4a9b-ae5e-0fcbadbde7ca 2024-09-18T15:14:52.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:14:52.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:14:52.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:14:52.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:14:52.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:14:52.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:14:52.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:14:52.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:14:52.132 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:14:52.133 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:14:52.133 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:14:52.133 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:14:52.133 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:14:52.133 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:14:52.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:14:52.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:14:52.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:14:52.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:14:52.364 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:14:52.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:14:52.703 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:14:52.703 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:14:52.703 INFO:tasks.workunit.client.0.smithi046.stderr:3' 2024-09-18T15:14:52.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:14:52.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:14:52.703 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:14:52.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836526 2024-09-18T15:14:52.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836526 2024-09-18T15:14:52.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836526' 2024-09-18T15:14:52.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:14:52.837 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:14:53.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=244813135874 2024-09-18T15:14:53.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 244813135874 2024-09-18T15:14:53.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836526 1-244813135874' 2024-09-18T15:14:53.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:14:53.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:14:53.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509470 2024-09-18T15:14:53.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509470 2024-09-18T15:14:53.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836526 1-244813135874 2-64424509470' 2024-09-18T15:14:53.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:14:53.157 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:14:53.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=214748364806 2024-09-18T15:14:53.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 214748364806 2024-09-18T15:14:53.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836526 1-244813135874 2-64424509470 3-214748364806' 2024-09-18T15:14:53.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:14:53.300 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836526 2024-09-18T15:14:53.300 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:14:53.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:14:53.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836526 2024-09-18T15:14:53.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:14:53.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836526 2024-09-18T15:14:53.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2024-09-18T15:14:53.306 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836526 2024-09-18T15:14:53.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:14:53.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836526 -lt 21474836526 2024-09-18T15:14:53.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:14:53.667 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-244813135874 2024-09-18T15:14:53.667 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:14:53.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:14:53.671 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-244813135874 2024-09-18T15:14:53.671 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:14:53.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=244813135874 2024-09-18T15:14:53.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 244813135874' 2024-09-18T15:14:53.674 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 244813135874 2024-09-18T15:14:53.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:14:54.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 244813135874 -lt 244813135874 2024-09-18T15:14:54.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:14:54.089 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509470 2024-09-18T15:14:54.089 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:14:54.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:14:54.092 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509470 2024-09-18T15:14:54.092 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:14:54.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509470 2024-09-18T15:14:54.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509470' 2024-09-18T15:14:54.095 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509470 2024-09-18T15:14:54.095 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:14:54.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509470 -lt 64424509470 2024-09-18T15:14:54.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:14:54.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-214748364806 2024-09-18T15:14:54.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:14:54.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:14:54.443 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-214748364806 2024-09-18T15:14:54.444 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:14:54.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=214748364806 2024-09-18T15:14:54.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 214748364806' 2024-09-18T15:14:54.446 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.3 seq 214748364806 2024-09-18T15:14:54.447 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:14:54.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 214748364806 -lt 214748364806 2024-09-18T15:14:54.759 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:14:54.760 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:14:54.760 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:14:55.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:14:55.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:14:55.177 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:14:55.177 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:14:55.177 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:14:55.177 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:14:55.177 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:14:55.177 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:14:55.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:14:55.518 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:14:55.518 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:14:55.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:14:55.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:14:55.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:14:55.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:14:55.883 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-18T15:14:55.883 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-09-18T15:14:55.883 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T15:14:55.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:14:55.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:14:56.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: local pg=2.0 2024-09-18T15:14:56.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: repair 2.0 2024-09-18T15:14:56.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-09-18T15:14:56.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-09-18T15:14:56.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:14:56.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:14:56.276 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:14:56.276 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:14:56.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T15:14:26.234869+0000 2024-09-18T15:14:56.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-09-18T15:14:56.897 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-18T15:14:56.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-09-18T15:14:26.234869+0000 2024-09-18T15:14:56.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=2.0 2024-09-18T15:14:56.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:14:26.234869+0000 2024-09-18T15:14:56.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:14:56.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:14:56.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:14:56.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:14:56.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:14:56.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:14:56.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:14:56.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:14:57.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:14:26.234869+0000 '>' 2024-09-18T15:14:26.234869+0000 2024-09-18T15:14:57.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:14:58.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:14:58.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:14:58.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:14:58.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:14:58.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:14:58.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:14:58.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:14:58.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:14:26.234869+0000 '>' 2024-09-18T15:14:26.234869+0000 2024-09-18T15:14:58.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:14:59.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:14:59.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:14:59.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:14:59.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:14:59.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:14:59.540 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:14:59.540 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:14:59.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:14:26.234869+0000 '>' 2024-09-18T15:14:26.234869+0000 2024-09-18T15:14:59.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:15:00.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:15:00.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:15:00.851 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:15:00.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:15:00.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:15:00.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:15:00.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:15:01.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:14:26.234869+0000 '>' 2024-09-18T15:14:26.234869+0000 2024-09-18T15:15:01.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:15:02.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:15:02.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:15:02.163 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:15:02.163 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:15:02.163 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:15:02.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:15:02.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:15:02.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:14:26.234869+0000 '>' 2024-09-18T15:14:26.234869+0000 2024-09-18T15:15:02.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:15:03.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:15:03.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:15:03.465 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:15:03.465 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:15:03.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:15:03.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:15:03.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:15:03.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:14:58.787111+0000 '>' 2024-09-18T15:14:26.234869+0000 2024-09-18T15:15:03.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:15:03.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:241: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:15:03.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:15:03.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:15:03.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-18T15:15:03.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:15:03.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:15:03.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:15:03.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:15:03.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-18T15:15:03.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:15:03.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-18T15:15:03.787 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:15:03.787 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:15:03.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:15:03.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:15:03.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:15:04.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:15:04.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:15:04.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:15:04.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:15:04.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-18T15:15:04.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:15:04.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:15:04.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-18T15:15:06.041 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-18T15:15:06.041 INFO:tasks.workunit.client.0.smithi046.stdout:_ 2024-09-18T15:15:06.041 INFO:tasks.workunit.client.0.smithi046.stdout:hinfo_key 2024-09-18T15:15:06.041 INFO:tasks.workunit.client.0.smithi046.stdout:snapset 2024-09-18T15:15:06.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-18T15:15:06.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:15:06.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:15:06.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T15:15:06.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:15:06.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:15:06.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:15:06.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:15:06.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:15:06.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:15:06.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-18T15:15:06.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-18T15:15:06.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:15:06.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:15:06.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:15:06.585 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:15:06.585 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:15:06.585 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:15:06.586 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:15:06.586 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:15:06.586 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:15:06.586 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:15:06.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:15:06.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:15:06.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:15:06.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:15:06.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:15:06.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:15:06.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:15:06.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:15:06.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:15:06.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:15:06.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-18T15:15:06.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T15:15:06.588 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.3 2024-09-18T15:15:06.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:15:06.590 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-18T15:15:06.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T15:15:06.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:15:06.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:15:06.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:15:06.619 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:15:06.611+0000 7f4205bbd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:15:06.623 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:15:06.615+0000 7f4205bbd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:15:06.628 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:15:06.619+0000 7f4205bbd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:15:06.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T15:15:06.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:15:06.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:15:06.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:15:06.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:15:06.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:06.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:15:06.908 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:15:06.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:06.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:15:07.159 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:15:07.151+0000 7f4205bbd540 -1 Falling back to public interface 2024-09-18T15:15:07.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:08.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:08.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:08.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:15:08.242 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:15:08.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:08.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:15:08.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:09.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:09.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:09.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:15:09.561 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:15:09.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:09.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:15:09.859 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:15:09.851+0000 7f4205bbd540 -1 osd.3 59 log_to_monitors true 2024-09-18T15:15:09.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:10.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:10.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:10.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:15:10.881 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:15:10.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:10.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:15:11.501 INFO:tasks.workunit.client.0.smithi046.stderr:osd.3 up in weight 1 up_from 64 up_thru 64 down_at 60 last_clean_interval [50,59) [v2:127.0.0.1:6826/241971,v1:127.0.0.1:6827/241971] [v2:127.0.0.1:6828/241971,v1:127.0.0.1:6829/241971] exists,up db129174-c3ee-42d9-b16d-41a610ac05c9 2024-09-18T15:15:11.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:15:11.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:15:11.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:15:11.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:15:11.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:15:11.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:15:11.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:15:11.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:15:11.503 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:15:11.504 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:15:11.504 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:15:11.505 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:15:11.505 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:15:11.505 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:15:11.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:15:11.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:15:11.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:15:11.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:15:11.726 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:15:12.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:15:12.261 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:15:12.261 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:15:12.261 INFO:tasks.workunit.client.0.smithi046.stderr:3' 2024-09-18T15:15:12.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:15:12.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:15:12.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:15:12.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836531 2024-09-18T15:15:12.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836531 2024-09-18T15:15:12.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836531' 2024-09-18T15:15:12.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:15:12.450 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:15:12.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=244813135879 2024-09-18T15:15:12.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 244813135879 2024-09-18T15:15:12.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836531 1-244813135879' 2024-09-18T15:15:12.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:15:12.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:15:12.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509475 2024-09-18T15:15:12.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509475 2024-09-18T15:15:12.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836531 1-244813135879 2-64424509475' 2024-09-18T15:15:12.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:15:12.735 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:15:12.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=274877906946 2024-09-18T15:15:12.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 274877906946 2024-09-18T15:15:12.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836531 1-244813135879 2-64424509475 3-274877906946' 2024-09-18T15:15:12.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:15:12.879 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836531 2024-09-18T15:15:12.879 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:15:12.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:15:12.882 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836531 2024-09-18T15:15:12.882 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:15:12.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836531 2024-09-18T15:15:12.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836531' 2024-09-18T15:15:12.883 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836531 2024-09-18T15:15:12.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:15:13.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836530 -lt 21474836531 2024-09-18T15:15:13.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:15:14.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:15:14.203 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:15:14.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836531 -lt 21474836531 2024-09-18T15:15:14.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:15:14.524 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-244813135879 2024-09-18T15:15:14.524 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:15:14.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:15:14.527 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-244813135879 2024-09-18T15:15:14.528 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:15:14.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=244813135879 2024-09-18T15:15:14.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 244813135879' 2024-09-18T15:15:14.530 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 244813135879 2024-09-18T15:15:14.531 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:15:14.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 244813135879 -lt 244813135879 2024-09-18T15:15:14.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:15:14.880 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509475 2024-09-18T15:15:14.880 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:15:14.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:15:14.883 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509475 2024-09-18T15:15:14.883 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:15:14.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509475 2024-09-18T15:15:14.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509475' 2024-09-18T15:15:14.886 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509475 2024-09-18T15:15:14.887 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:15:15.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509475 -lt 64424509475 2024-09-18T15:15:15.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:15:15.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-274877906946 2024-09-18T15:15:15.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:15:15.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:15:15.295 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-274877906946 2024-09-18T15:15:15.295 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:15:15.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=274877906946 2024-09-18T15:15:15.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 274877906946' 2024-09-18T15:15:15.297 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.3 seq 274877906946 2024-09-18T15:15:15.297 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:15:15.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 274877906946 -lt 274877906946 2024-09-18T15:15:15.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:15:15.637 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:15:15.637 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:15:16.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:15:16.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:15:16.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:15:16.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:15:16.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:15:16.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:15:16.055 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:15:16.055 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:15:16.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:15:16.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:15:16.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:15:16.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:15:16.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:15:16.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:15:16.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:15:16.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-18T15:15:16.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-18T15:15:16.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-09-18T15:15:16.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:184: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-18T15:15:16.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local poolname=ecpool 2024-09-18T15:15:16.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local first=1 2024-09-18T15:15:16.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local second=2 2024-09-18T15:15:16.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:192: corrupt_and_repair_two: pids= 2024-09-18T15:15:16.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:15:16.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:15:16.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:15:16.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 242988"' 2024-09-18T15:15:16.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 242988' 2024-09-18T15:15:16.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-18T15:15:16.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:15:16.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:15:16.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 242991"' 2024-09-18T15:15:16.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/242990: /' 2024-09-18T15:15:16.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 242991' 2024-09-18T15:15:16.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: wait_background pids 2024-09-18T15:15:16.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: wait_background: pids=' 242988 242991' 2024-09-18T15:15:16.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: wait_background: return_code=0 2024-09-18T15:15:16.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:15:16.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 242988 2024-09-18T15:15:16.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:15:16.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-18T15:15:16.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/242993: /' 2024-09-18T15:15:19.813 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:15:19.813 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:15:19.813 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:15:19.813 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:15:19.813 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:15:19.813 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:15:19.813 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:15:19.813 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:15:19.814 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:15:19.814 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:15:19.814 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:15:19.814 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:15:19.814 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:15:19.814 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:15:19.814 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:15:19.814 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:15:19.814 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:15:19.814 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:15:19.814 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:15:19.814 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:15:19.814 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:15:19.815 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:15:19.815 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-18T15:15:19.815 INFO:tasks.workunit.client.0.smithi046.stderr:242990: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-18T15:15:19.815 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:15:19.815 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:15:19.815 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:15:19.815 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:15:19.815 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:15:19.815 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:15:19.816 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:15:19.816 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:15:19.817 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:15:19.817 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:15:19.817 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:15:19.817 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:15:19.818 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:15:19.818 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:15:19.818 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:15:19.818 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:15:19.818 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:15:19.818 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:15:19.818 INFO:tasks.workunit.client.0.smithi046.stderr:242990: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:15:19.818 INFO:tasks.workunit.client.0.smithi046.stderr:242990: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:15:19.818 INFO:tasks.workunit.client.0.smithi046.stderr:242990: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:15:19.818 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:15:19.818 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:15:19.818 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:15:19.819 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:15:19.819 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:15:19.819 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:15:19.819 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:15:19.819 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:15:19.819 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:15:19.819 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:15:19.819 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:15:19.819 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:15:19.819 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:15:19.819 INFO:tasks.workunit.client.0.smithi046.stderr:242990: start osd.1 2024-09-18T15:15:20.022 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:15:20.022 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:15:20.022 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-18T15:15:20.022 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:15:20.022 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-09-18T15:15:20.023 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:15:20.023 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:15:20.023 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-18T15:15:20.023 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:15:20.023 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-18T15:15:20.023 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:15:20.023 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:15:20.023 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:15:20.023 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:15:20.023 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:15:20.024 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:15:20.024 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-09-18T15:15:20.024 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:15:20.024 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:15:20.024 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-18T15:15:20.024 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:15:20.024 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:15:20.024 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-09-18T15:15:20.024 INFO:tasks.workunit.client.0.smithi046.stderr:242993: remove 2#2:eb822e21:::SOMETHING:head# 2024-09-18T15:15:20.024 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-18T15:15:20.024 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:15:20.024 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:15:20.024 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T15:15:20.025 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:15:20.025 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:15:20.025 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:15:20.025 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:15:20.028 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:15:20.028 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:15:20.028 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T15:15:20.028 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T15:15:20.028 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:15:20.029 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:15:20.029 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:15:20.029 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:15:20.029 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:15:20.029 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:15:20.029 INFO:tasks.workunit.client.0.smithi046.stderr:242993: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:15:20.029 INFO:tasks.workunit.client.0.smithi046.stderr:242993: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:15:20.029 INFO:tasks.workunit.client.0.smithi046.stderr:242993: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:15:20.029 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:15:20.029 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:15:20.029 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:15:20.030 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:15:20.030 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:15:20.030 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:15:20.030 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:15:20.030 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:15:20.030 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:15:20.030 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:15:20.030 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:15:20.030 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-18T15:15:20.030 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T15:15:20.030 INFO:tasks.workunit.client.0.smithi046.stderr:242993: start osd.2 2024-09-18T15:15:24.575 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:15:24.575 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:15:24.575 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:15:24.575 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:15:24.575 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:15:24.576 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:15:24.576 INFO:tasks.workunit.client.0.smithi046.stderr:242990: 2024-09-18T15:15:19.843+0000 7fa8cb873540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:15:24.576 INFO:tasks.workunit.client.0.smithi046.stderr:242990: 2024-09-18T15:15:19.847+0000 7fa8cb873540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:15:24.576 INFO:tasks.workunit.client.0.smithi046.stderr:242990: 2024-09-18T15:15:19.847+0000 7fa8cb873540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:15:24.576 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:15:24.576 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:15:24.576 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:15:24.576 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:15:24.576 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:15:24.576 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:24.576 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:15:24.576 INFO:tasks.workunit.client.0.smithi046.stderr:242990: 0 2024-09-18T15:15:24.576 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:24.576 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:15:24.576 INFO:tasks.workunit.client.0.smithi046.stderr:242990: 2024-09-18T15:15:20.395+0000 7fa8cb873540 -1 Falling back to public interface 2024-09-18T15:15:24.577 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:24.577 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:24.577 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:24.577 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:15:24.577 INFO:tasks.workunit.client.0.smithi046.stderr:242990: 1 2024-09-18T15:15:24.577 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:24.577 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:15:24.577 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:24.577 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:24.577 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:24.577 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:15:24.577 INFO:tasks.workunit.client.0.smithi046.stderr:242990: 2 2024-09-18T15:15:24.577 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:24.577 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:15:24.577 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:24.578 INFO:tasks.workunit.client.0.smithi046.stderr:242990: 2024-09-18T15:15:23.607+0000 7fa8cb873540 -1 osd.1 66 log_to_monitors true 2024-09-18T15:15:24.578 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:24.578 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:24.578 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:15:24.578 INFO:tasks.workunit.client.0.smithi046.stderr:242990: 3 2024-09-18T15:15:24.578 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:24.578 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:15:24.702 INFO:tasks.workunit.client.0.smithi046.stderr:242990: osd.1 up in weight 1 up_from 70 up_thru 60 down_at 67 lops 2024-09-18T15:15:24.702 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-18T15:15:24.702 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T15:15:24.702 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:15:24.702 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:15:24.702 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:15:24.702 INFO:tasks.workunit.client.0.smithi046.stderr:242993: 2024-09-18T15:15:20.051+0000 7f84739ac540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:15:24.702 INFO:tasks.workunit.client.0.smithi046.stderr:242993: 2024-09-18T15:15:20.055+0000 7f84739ac540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:15:24.703 INFO:tasks.workunit.client.0.smithi046.stderr:242993: 2024-09-18T15:15:20.059+0000 7f84739ac540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:15:24.703 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T15:15:24.703 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:15:24.703 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:15:24.703 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:15:24.703 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:15:24.703 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:24.703 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:15:24.703 INFO:tasks.workunit.client.0.smithi046.stderr:242993: 0 2024-09-18T15:15:24.703 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:24.704 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:15:24.704 INFO:tasks.workunit.client.0.smithi046.stderr:242993: 2024-09-18T15:15:20.355+0000 7f84739ac540 -1 Falling back to public interface 2024-09-18T15:15:24.704 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:24.704 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:24.704 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:24.704 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:15:24.704 INFO:tasks.workunit.client.0.smithi046.stderr:242993: 1 2024-09-18T15:15:24.704 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:24.704 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:15:24.704 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:24.704 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:24.704 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:24.704 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:15:24.705 INFO:tasks.workunit.client.0.smithi046.stderr:242993: 2 2024-09-18T15:15:24.705 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:24.705 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:15:24.705 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:24.705 INFO:tasks.workunit.client.0.smithi046.stderr:242993: 2024-09-18T15:15:23.707+0000 7f84739ac540 -1 osd.2 66 log_to_monitors true 2024-09-18T15:15:24.705 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:24.705 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:24.705 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:15:24.705 INFO:tasks.workunit.client.0.smithi046.stderr:242993: 3 2024-09-18T15:15:24.705 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:24.705 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:15:25.658 INFO:tasks.workunit.client.0.smithi046.stderr:242993: osd.2 up in weight 1 up_from 70 up_thru 53 down_at 67 last_clean_interval [57,66) [v2:127.0.0.1:6818/243798,v1:127.0.0.1:6819/243798] [v2:127.0.0.1:6820/243798,v1:127.0.0.1:6821/243798] exists,up b59f73a0-2db8-4a9b-ae5e-0fcbadbde7ca 2024-09-18T15:15:25.659 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:15:25.659 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:15:25.659 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:15:25.659 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:15:25.659 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:15:25.659 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:15:25.659 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:15:25.659 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:15:25.659 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:15:25.659 INFO:tasks.workunit.client.0.smithi046.stderr:242990: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:15:25.659 INFO:tasks.workunit.client.0.smithi046.stderr:242990: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:15:25.660 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:15:25.660 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:15:25.660 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:15:25.660 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:15:25.660 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:15:25.660 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:15:25.660 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:15:25.660 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:15:25.660 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:15:25.660 INFO:tasks.workunit.client.0.smithi046.stderr:242990: 1 2024-09-18T15:15:25.660 INFO:tasks.workunit.client.0.smithi046.stderr:242990: 2 2024-09-18T15:15:25.660 INFO:tasks.workunit.client.0.smithi046.stderr:242990: 3' 2024-09-18T15:15:25.661 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:15:25.661 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:15:25.661 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:15:25.661 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836535 2024-09-18T15:15:25.661 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836535 2024-09-18T15:15:25.661 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836535' 2024-09-18T15:15:25.661 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:15:25.661 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:15:25.661 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=300647710722 2024-09-18T15:15:25.661 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 300647710722 2024-09-18T15:15:25.661 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836535 1-300647710722' 2024-09-18T15:15:25.661 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:15:25.661 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:15:25.993 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /homast_clean_interval [15,66) [v2:127.0.0.1:6810/243843,v1:127.0.0.1:6811/243843] [v2:127.0.0.1:6812/243843,v1:127.0.0.1:6813/243843] exists,up af350427-c715-418c-bd46-5204bfa0c41e 2024-09-18T15:15:25.993 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:15:25.993 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:15:25.993 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:15:25.993 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:15:25.993 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:15:25.994 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:15:25.994 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:15:25.994 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:15:25.994 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:15:25.994 INFO:tasks.workunit.client.0.smithi046.stderr:242993: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:15:25.994 INFO:tasks.workunit.client.0.smithi046.stderr:242993: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:15:25.994 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:15:25.994 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:15:25.994 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:15:25.994 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:15:25.994 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:15:25.994 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:15:25.994 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:15:25.994 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:15:25.994 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:15:25.995 INFO:tasks.workunit.client.0.smithi046.stderr:242993: 1 2024-09-18T15:15:25.995 INFO:tasks.workunit.client.0.smithi046.stderr:242993: 2 2024-09-18T15:15:25.995 INFO:tasks.workunit.client.0.smithi046.stderr:242993: 3' 2024-09-18T15:15:25.995 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:15:25.995 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:15:25.995 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:15:25.995 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836536 2024-09-18T15:15:25.995 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836536 2024-09-18T15:15:25.995 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836536' 2024-09-18T15:15:25.995 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:15:25.995 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:15:25.995 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=300647710723 2024-09-18T15:15:25.995 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 300647710723 2024-09-18T15:15:25.995 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836536 1-300647710723' 2024-09-18T15:15:25.996 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:15:25.996 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:15:28.064 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=300647710722 2024-09-18T15:15:28.064 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 300647710722 2024-09-18T15:15:28.064 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836535 1-300647710722 2-300647710722' 2024-09-18T15:15:28.064 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:15:28.065 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:15:28.065 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=274877906950 2024-09-18T15:15:28.065 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 274877906950 2024-09-18T15:15:28.065 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836535 1-300647710722 2-300647710722 3-274877906950' 2024-09-18T15:15:28.065 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:15:28.065 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836535 2024-09-18T15:15:28.065 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:15:28.065 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:15:28.065 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836535 2024-09-18T15:15:28.065 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:15:28.065 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836535 2024-09-18T15:15:28.066 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836535' 2024-09-18T15:15:28.066 INFO:tasks.workunit.client.0.smithi046.stderr:242990: waiting osd.0 seq 21474836535 2024-09-18T15:15:28.066 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:15:28.066 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836535 -lt 21474836535 2024-09-18T15:15:28.066 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:15:28.066 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-300647710722 2024-09-18T15:15:28.066 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:15:28.066 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:15:28.066 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-300647710722 2024-09-18T15:15:28.066 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:15:28.066 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=300647710722 2024-09-18T15:15:28.066 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 300647710722' 2024-09-18T15:15:28.066 INFO:tasks.workunit.client.0.smithi046.stderr:242990: waiting osd.1 seq 300647710722 2024-09-18T15:15:28.067 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:15:28.067 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 300647710722 2024-09-18T15:15:28.067 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:15:28.067 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:15:28.067 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:15:28.067 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 300647710723 -lt 300647710722 2024-09-18T15:15:28.067 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:15:28.295 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/ste/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=300647710723 2024-09-18T15:15:28.295 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 300647710723 2024-09-18T15:15:28.295 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836536 1-300647710723 2-300647710723' 2024-09-18T15:15:28.295 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:15:28.295 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:15:28.295 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=274877906951 2024-09-18T15:15:28.295 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 274877906951 2024-09-18T15:15:28.295 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836536 1-300647710723 2-300647710723 3-274877906951' 2024-09-18T15:15:28.295 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:15:28.295 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836536 2024-09-18T15:15:28.296 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:15:28.296 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:15:28.296 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836536 2024-09-18T15:15:28.296 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:15:28.296 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836536 2024-09-18T15:15:28.296 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836536' 2024-09-18T15:15:28.296 INFO:tasks.workunit.client.0.smithi046.stderr:242993: waiting osd.0 seq 21474836536 2024-09-18T15:15:28.296 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:15:28.296 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836535 -lt 21474836536 2024-09-18T15:15:28.296 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:15:28.296 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:15:28.296 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:15:28.297 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836536 -lt 21474836536 2024-09-18T15:15:28.297 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:15:28.297 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-300647710723 2024-09-18T15:15:28.297 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:15:28.297 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:15:28.297 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-300647710723 2024-09-18T15:15:28.297 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:15:28.297 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=300647710723 2024-09-18T15:15:28.297 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 300647710723' 2024-09-18T15:15:28.297 INFO:tasks.workunit.client.0.smithi046.stderr:242993: waiting osd.1 seq 300647710723 2024-09-18T15:15:28.297 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:15:28.297 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 300647710723 -lt 300647710723 2024-09-18T15:15:28.297 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:15:29.641 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.clienandalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-300647710722 2024-09-18T15:15:29.641 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:15:29.641 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:15:29.641 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-300647710722 2024-09-18T15:15:29.641 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:15:29.642 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=300647710722 2024-09-18T15:15:29.642 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 300647710722' 2024-09-18T15:15:29.642 INFO:tasks.workunit.client.0.smithi046.stderr:242990: waiting osd.2 seq 300647710722 2024-09-18T15:15:29.642 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:15:29.642 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 300647710723 -lt 300647710722 2024-09-18T15:15:29.642 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:15:29.642 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-274877906950 2024-09-18T15:15:29.642 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:15:29.642 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:15:29.642 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:15:29.643 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-274877906950 2024-09-18T15:15:29.643 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=274877906950 2024-09-18T15:15:29.643 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 274877906950' 2024-09-18T15:15:29.643 INFO:tasks.workunit.client.0.smithi046.stderr:242990: waiting osd.3 seq 274877906950 2024-09-18T15:15:29.643 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:15:29.643 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 274877906951 -lt 274877906950 2024-09-18T15:15:29.643 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:15:29.643 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:15:29.643 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:15:29.643 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:15:29.644 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:15:29.644 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:15:29.644 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:15:29.644 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:15:29.644 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:15:29.644 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:15:29.644 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:15:29.644 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T15:15:29.644 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:15:29.764 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: gett.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-300647710723 2024-09-18T15:15:29.764 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:15:29.764 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:15:29.764 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-300647710723 2024-09-18T15:15:29.764 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:15:29.765 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=300647710723 2024-09-18T15:15:29.765 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 300647710723' 2024-09-18T15:15:29.765 INFO:tasks.workunit.client.0.smithi046.stderr:242993: waiting osd.2 seq 300647710723 2024-09-18T15:15:29.765 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:15:29.765 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 300647710723 -lt 300647710723 2024-09-18T15:15:29.765 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:15:29.765 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-274877906951 2024-09-18T15:15:29.765 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:15:29.765 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:15:29.765 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-274877906951 2024-09-18T15:15:29.765 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:15:29.765 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=274877906951 2024-09-18T15:15:29.765 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 274877906951' 2024-09-18T15:15:29.766 INFO:tasks.workunit.client.0.smithi046.stderr:242993: waiting osd.3 seq 274877906951 2024-09-18T15:15:29.766 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:15:29.766 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 274877906951 -lt 274877906951 2024-09-18T15:15:29.766 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:15:29.766 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:15:29.766 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:15:29.766 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:15:29.766 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:15:29.766 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:15:29.766 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:15:29.766 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:15:29.766 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:15:29.767 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:15:29.767 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:15:29.767 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T15:15:29.767 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:15:31.237 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:_num_pgs: ceph --format json status 2024-09-18T15:15:31.237 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:15:31.237 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-09-18T15:15:31.237 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-09-18T15:15:31.237 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T15:15:31.238 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-09-18T15:15:31.238 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:15:31.238 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:15:31.238 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:15:31.238 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:15:31.238 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:15:31.238 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:15:31.238 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:15:31.238 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:15:31.238 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:15:31.238 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:15:31.238 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T15:15:31.238 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:15:31.238 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:15:31.239 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:15:31.239 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-09-18T15:15:31.239 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-09-18T15:15:31.239 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:15:31.239 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:15:31.239 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:15:31.239 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:15:31.239 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:15:31.239 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:15:31.239 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:15:31.239 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T15:15:31.239 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T15:15:31.400 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:161425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:15:31.401 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:15:31.401 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-09-18T15:15:31.401 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-09-18T15:15:31.401 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T15:15:31.401 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-09-18T15:15:31.401 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:15:31.401 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:15:31.401 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:15:31.401 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:15:31.401 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:15:31.401 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:15:31.401 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:15:31.402 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:15:31.402 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:15:31.402 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:15:31.402 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T15:15:31.402 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:15:31.402 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:15:31.402 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:15:31.402 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-09-18T15:15:31.402 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-09-18T15:15:31.402 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:15:31.402 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:15:31.402 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:15:31.402 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:15:31.403 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:15:31.403 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:15:31.403 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:15:31.403 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T15:15:31.403 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T15:15:32.180 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-help64: wait_for_clean: (( 1 >= 13 )) 2024-09-18T15:15:32.180 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:15:32.180 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-18T15:15:32.180 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:15:32.180 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:15:32.180 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:15:32.180 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:15:32.181 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:15:32.181 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:15:32.181 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:15:32.181 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:15:32.181 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:15:32.181 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:15:32.181 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:15:32.181 INFO:tasks.workunit.client.0.smithi046.stderr:242990: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:15:32.181 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:15:32.181 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:15:32.181 INFO:tasks.workunit.client.0.smithi046.stderr:242990: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:15:32.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:15:32.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:15:32.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 242991 2024-09-18T15:15:32.295 INFO:tasks.workunit.client.0.smithi046.stderr:ers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-09-18T15:15:32.295 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:15:32.295 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-18T15:15:32.295 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:15:32.295 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:15:32.295 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:15:32.295 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:15:32.295 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:15:32.296 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:15:32.296 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:15:32.296 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:15:32.296 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:15:32.296 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:15:32.296 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:15:32.296 INFO:tasks.workunit.client.0.smithi046.stderr:242993: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:15:32.296 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:15:32.296 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:15:32.296 INFO:tasks.workunit.client.0.smithi046.stderr:242993: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:15:32.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:15:32.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: eval 'pids='\'''\''' 2024-09-18T15:15:32.298 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: pids= 2024-09-18T15:15:32.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: wait_background: return 0 2024-09-18T15:15:32.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: return_code=0 2024-09-18T15:15:32.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-18T15:15:32.298 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-18T15:15:32.298 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-09-18T15:15:32.298 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T15:15:32.298 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:15:32.298 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:15:32.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: local pg=2.0 2024-09-18T15:15:32.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: repair 2.0 2024-09-18T15:15:32.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-09-18T15:15:32.640 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-09-18T15:15:32.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:15:32.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:15:32.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:15:32.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:15:32.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T15:14:58.787111+0000 2024-09-18T15:15:32.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-09-18T15:15:33.294 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-18T15:15:33.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-09-18T15:14:58.787111+0000 2024-09-18T15:15:33.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=2.0 2024-09-18T15:15:33.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:14:58.787111+0000 2024-09-18T15:15:33.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:15:33.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:15:33.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:15:33.308 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:15:33.308 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:15:33.308 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:15:33.308 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:15:33.308 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:15:33.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:14:58.787111+0000 '>' 2024-09-18T15:14:58.787111+0000 2024-09-18T15:15:33.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:15:34.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:15:34.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:15:34.620 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:15:34.620 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:15:34.620 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:15:34.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:15:34.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:15:34.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:14:58.787111+0000 '>' 2024-09-18T15:14:58.787111+0000 2024-09-18T15:15:34.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:15:35.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:15:35.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:15:35.936 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:15:35.936 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:15:35.937 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:15:35.937 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:15:35.937 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:15:36.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:14:58.787111+0000 '>' 2024-09-18T15:14:58.787111+0000 2024-09-18T15:15:36.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:15:37.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:15:37.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:15:37.258 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:15:37.258 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:15:37.258 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:15:37.259 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:15:37.259 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:15:37.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:15:34.131205+0000 '>' 2024-09-18T15:14:58.787111+0000 2024-09-18T15:15:37.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:15:37.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:207: corrupt_and_repair_two: pids= 2024-09-18T15:15:37.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:15:37.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:15:37.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:15:37.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 246236"' 2024-09-18T15:15:37.569 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 246236' 2024-09-18T15:15:37.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-18T15:15:37.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:15:37.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:15:37.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:15:37.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/246238: /' 2024-09-18T15:15:37.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 246239"' 2024-09-18T15:15:37.571 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 246239' 2024-09-18T15:15:37.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: wait_background pids 2024-09-18T15:15:37.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: wait_background: pids=' 246236 246239' 2024-09-18T15:15:37.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: wait_background: return_code=0 2024-09-18T15:15:37.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:15:37.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 246236 2024-09-18T15:15:37.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-18T15:15:37.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/246242: /' 2024-09-18T15:15:41.703 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:15:41.703 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:15:41.703 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:15:41.703 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:15:41.703 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:15:41.703 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:15:41.703 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:15:41.703 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:15:41.703 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:15:41.703 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:15:41.703 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:15:41.703 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:15:41.704 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:15:41.704 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:15:41.704 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:15:41.704 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:15:41.704 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:15:41.704 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:15:41.704 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:15:41.704 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:15:41.704 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:15:41.704 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:15:41.704 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-18T15:15:41.704 INFO:tasks.workunit.client.0.smithi046.stderr:246238: _ 2024-09-18T15:15:41.704 INFO:tasks.workunit.client.0.smithi046.stderr:246238: hinfo_key 2024-09-18T15:15:41.704 INFO:tasks.workunit.client.0.smithi046.stderr:246238: snapset 2024-09-18T15:15:41.705 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:15:41.705 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:15:41.705 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:15:41.705 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:15:41.705 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:15:41.705 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:15:41.705 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:15:41.705 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:15:41.709 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:15:41.709 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:15:41.709 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:15:41.709 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:15:41.709 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:15:41.709 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:15:41.709 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:15:41.710 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:15:41.710 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:15:41.710 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:15:41.710 INFO:tasks.workunit.client.0.smithi046.stderr:246238: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:15:41.710 INFO:tasks.workunit.client.0.smithi046.stderr:246238: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:15:41.710 INFO:tasks.workunit.client.0.smithi046.stderr:246238: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:15:41.710 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:15:41.710 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:15:41.710 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:15:41.710 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:15:41.710 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:15:41.710 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:15:41.710 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:15:41.710 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:15:41.711 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:15:41.711 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:15:41.711 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:15:41.711 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:15:41.711 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:15:41.711 INFO:tasks.workunit.client.0.smithi046.stderr:246238: start osd.1 2024-09-18T15:15:41.866 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:15:41.866 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:15:41.866 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-18T15:15:41.866 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:15:41.866 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-18T15:15:41.866 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:15:41.866 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:15:41.866 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-18T15:15:41.867 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:15:41.867 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-18T15:15:41.867 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:15:41.867 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:15:41.867 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:15:41.867 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:15:41.867 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:15:41.867 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:15:41.867 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-18T15:15:41.867 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:15:41.867 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:15:41.867 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-18T15:15:41.867 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:15:41.868 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:15:41.868 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-09-18T15:15:41.868 INFO:tasks.workunit.client.0.smithi046.stderr:246242: _ 2024-09-18T15:15:41.868 INFO:tasks.workunit.client.0.smithi046.stderr:246242: hinfo_key 2024-09-18T15:15:41.868 INFO:tasks.workunit.client.0.smithi046.stderr:246242: snapset 2024-09-18T15:15:41.868 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-18T15:15:41.868 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:15:41.868 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:15:41.868 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T15:15:41.868 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:15:41.868 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:15:41.869 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:15:41.869 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:15:41.871 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:15:41.871 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:15:41.871 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T15:15:41.871 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T15:15:41.871 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:15:41.872 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:15:41.872 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:15:41.872 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:15:41.872 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:15:41.872 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:15:41.872 INFO:tasks.workunit.client.0.smithi046.stderr:246242: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:15:41.872 INFO:tasks.workunit.client.0.smithi046.stderr:246242: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:15:41.872 INFO:tasks.workunit.client.0.smithi046.stderr:246242: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:15:41.872 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:15:41.873 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:15:41.873 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:15:41.873 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:15:41.873 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:15:41.873 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:15:41.873 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:15:41.873 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:15:41.873 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:15:41.873 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:15:41.873 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:15:41.873 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-18T15:15:41.873 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T15:15:41.873 INFO:tasks.workunit.client.0.smithi046.stderr:246242: start osd.2 2024-09-18T15:15:46.577 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:15:46.577 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-18T15:15:46.577 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T15:15:46.577 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:15:46.577 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:15:46.577 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:15:46.577 INFO:tasks.workunit.client.0.smithi046.stderr:246242: 2024-09-18T15:15:41.890+0000 7fde35b0c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:15:46.577 INFO:tasks.workunit.client.0.smithi046.stderr:246242: 2024-09-18T15:15:41.894+0000 7fde35b0c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:15:46.577 INFO:tasks.workunit.client.0.smithi046.stderr:246242: 2024-09-18T15:15:41.898+0000 7fde35b0c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:15:46.577 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T15:15:46.578 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:15:46.578 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:15:46.578 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:15:46.578 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:15:46.578 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:46.578 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:15:46.578 INFO:tasks.workunit.client.0.smithi046.stderr:246242: 0 2024-09-18T15:15:46.578 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:46.578 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:15:46.578 INFO:tasks.workunit.client.0.smithi046.stderr:246242: 2024-09-18T15:15:42.470+0000 7fde35b0c540 -1 Falling back to public interface 2024-09-18T15:15:46.578 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:46.578 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:46.578 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:46.578 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:15:46.579 INFO:tasks.workunit.client.0.smithi046.stderr:246242: 1 2024-09-18T15:15:46.579 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:46.579 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:15:46.579 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:46.579 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:46.579 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:46.579 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:15:46.579 INFO:tasks.workunit.client.0.smithi046.stderr:246242: 2 2024-09-18T15:15:46.579 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:46.579 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:15:46.579 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:46.579 INFO:tasks.workunit.client.0.smithi046.stderr:246242: 2024-09-18T15:15:45.750+0000 7fde35b0c540 -1 osd.2 72 log_to_monitors true 2024-09-18T15:15:46.579 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:46.579 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:46.580 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:15:46.580 INFO:tasks.workunit.client.0.smithi046.stderr:246242: 3 2024-09-18T15:15:46.580 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:46.580 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:15:46.580 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalo_recovery_ops 2024-09-18T15:15:46.580 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:15:46.580 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:15:46.580 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:15:46.580 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:15:46.581 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:15:46.581 INFO:tasks.workunit.client.0.smithi046.stderr:246238: 2024-09-18T15:15:41.730+0000 7ffbf5d1e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:15:46.581 INFO:tasks.workunit.client.0.smithi046.stderr:246238: 2024-09-18T15:15:41.734+0000 7ffbf5d1e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:15:46.581 INFO:tasks.workunit.client.0.smithi046.stderr:246238: 2024-09-18T15:15:41.738+0000 7ffbf5d1e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:15:46.581 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:15:46.581 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:15:46.581 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:15:46.581 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:15:46.581 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:15:46.581 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:46.581 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:15:46.581 INFO:tasks.workunit.client.0.smithi046.stderr:246238: 0 2024-09-18T15:15:46.581 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:46.582 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:15:46.582 INFO:tasks.workunit.client.0.smithi046.stderr:246238: 2024-09-18T15:15:42.286+0000 7ffbf5d1e540 -1 Falling back to public interface 2024-09-18T15:15:46.582 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:46.582 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:46.582 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:46.582 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:15:46.582 INFO:tasks.workunit.client.0.smithi046.stderr:246238: 1 2024-09-18T15:15:46.582 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:46.582 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:15:46.582 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:46.582 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:46.582 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:46.582 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:15:46.582 INFO:tasks.workunit.client.0.smithi046.stderr:246238: 2 2024-09-18T15:15:46.583 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:46.583 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:15:46.583 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:46.583 INFO:tasks.workunit.client.0.smithi046.stderr:246238: 2024-09-18T15:15:45.134+0000 7ffbf5d1e540 -1 osd.1 72 log_to_monitors true 2024-09-18T15:15:46.583 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:46.583 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:46.583 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:15:46.583 INFO:tasks.workunit.client.0.smithi046.stderr:246238: 3 2024-09-18T15:15:46.583 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:46.583 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:15:47.782 INFO:tasks.workunit.client.0.smithi046.stderr:246238: osd.1 up in weight 1 up_from 76 up_thru 70 down_at 73 last_clean_interval [70,72) [v2:127.0.0.1:6810/247048,v1:127.0.0.1:6811/247048] [v2:127.0.0.1:6812/247048,v1:127.0.0.1:6813/247048] exists,up b59f73a0-2db8-4a9b-ae5e-0fcbadbde7ca 2024-09-18T15:15:47.782 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:15:47.782 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:15:47.782 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:15:47.782 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:15:47.782 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:15:47.782 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:15:47.783 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:15:47.783 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:15:47.783 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:15:47.783 INFO:tasks.workunit.client.0.smithi046.stderr:246238: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:15:47.783 INFO:tasks.workunit.client.0.smithi046.stderr:246238: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:15:47.783 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:15:47.783 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:15:47.783 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:15:47.783 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:15:47.783 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:15:47.783 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:15:47.783 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:15:47.783 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:15:47.783 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:15:47.784 INFO:tasks.workunit.client.0.smithi046.stderr:246238: 1 2024-09-18T15:15:47.784 INFO:tasks.workunit.client.0.smithi046.stderr:246238: 2 2024-09-18T15:15:47.784 INFO:tasks.workunit.client.0.smithi046.stderr:246238: 3' 2024-09-18T15:15:47.784 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:15:47.784 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:15:47.784 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:15:47.784 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836541 2024-09-18T15:15:47.784 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836541 2024-09-18T15:15:47.784 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836541' 2024-09-18T15:15:47.784 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:15:47.784 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:15:47.784 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=326417514498 2024-09-18T15:15:47.784 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 326417514498 2024-09-18T15:15:47.784 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836541 1-326417514498' 2024-09-18T15:15:47.785 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:15:47.785 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:15:48.750 INFO:tasks.workunit.client.0.smithi046.stderr:24ne/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:15:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:246242: 4 2024-09-18T15:15:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:15:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:246242: osd.2 up in weight 1 up_from 79 up_thru 53 down_at 73 last_clean_interval [70,72) [v2:127.0.0.1:6818/247093,v1:127.0.0.1:6819/247093] [v2:127.0.0.1:6820/247093,v1:127.0.0.1:6821/247093] exists,up af350427-c715-418c-bd46-5204bfa0c41e 2024-09-18T15:15:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:15:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:15:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:15:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:15:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:15:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:15:48.752 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:15:48.752 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:15:48.752 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:15:48.752 INFO:tasks.workunit.client.0.smithi046.stderr:246242: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:15:48.752 INFO:tasks.workunit.client.0.smithi046.stderr:246242: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:15:48.752 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:15:48.752 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:15:48.752 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:15:48.752 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:15:48.752 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:15:48.752 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:15:48.752 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:15:48.752 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:15:48.752 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:15:48.753 INFO:tasks.workunit.client.0.smithi046.stderr:246242: 1 2024-09-18T15:15:48.753 INFO:tasks.workunit.client.0.smithi046.stderr:246242: 2 2024-09-18T15:15:48.753 INFO:tasks.workunit.client.0.smithi046.stderr:246242: 3' 2024-09-18T15:15:48.753 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:15:48.753 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:15:48.753 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:15:48.753 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836542 2024-09-18T15:15:48.753 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836542 2024-09-18T15:15:48.753 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836542' 2024-09-18T15:15:48.753 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:15:50.125 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats6238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=339302416386 2024-09-18T15:15:50.126 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 339302416386 2024-09-18T15:15:50.126 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836541 1-326417514498 2-339302416386' 2024-09-18T15:15:50.126 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:15:50.126 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:15:50.126 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=274877906956 2024-09-18T15:15:50.126 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 274877906956 2024-09-18T15:15:50.126 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836541 1-326417514498 2-339302416386 3-274877906956' 2024-09-18T15:15:50.126 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:15:50.126 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836541 2024-09-18T15:15:50.126 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:15:50.126 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:15:50.126 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836541 2024-09-18T15:15:50.126 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:15:50.126 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836541 2024-09-18T15:15:50.127 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836541' 2024-09-18T15:15:50.127 INFO:tasks.workunit.client.0.smithi046.stderr:246238: waiting osd.0 seq 21474836541 2024-09-18T15:15:50.127 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:15:50.127 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836541 -lt 21474836541 2024-09-18T15:15:50.127 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:15:50.127 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-326417514498 2024-09-18T15:15:50.127 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:15:50.127 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:15:50.127 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-326417514498 2024-09-18T15:15:50.127 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:15:50.127 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=326417514498 2024-09-18T15:15:50.127 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 326417514498' 2024-09-18T15:15:50.127 INFO:tasks.workunit.client.0.smithi046.stderr:246238: waiting osd.1 seq 326417514498 2024-09-18T15:15:50.127 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:15:50.127 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 326417514498 2024-09-18T15:15:50.128 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:15:50.128 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:15:50.128 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:15:50.128 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 326417514499 -lt 326417514498 2024-09-18T15:15:50.128 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:15:51.135 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.clie: ceph tell osd.1 flush_pg_stats 2024-09-18T15:15:51.136 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=326417514499 2024-09-18T15:15:51.136 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 326417514499 2024-09-18T15:15:51.136 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836542 1-326417514499' 2024-09-18T15:15:51.136 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:15:51.136 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:15:51.136 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=339302416387 2024-09-18T15:15:51.136 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 339302416387 2024-09-18T15:15:51.137 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836542 1-326417514499 2-339302416387' 2024-09-18T15:15:51.137 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:15:51.137 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:15:51.137 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=274877906957 2024-09-18T15:15:51.137 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 274877906957 2024-09-18T15:15:51.137 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836542 1-326417514499 2-339302416387 3-274877906957' 2024-09-18T15:15:51.137 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:15:51.137 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836542 2024-09-18T15:15:51.137 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:15:51.137 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:15:51.137 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836542 2024-09-18T15:15:51.137 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:15:51.137 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836542 2024-09-18T15:15:51.137 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836542' 2024-09-18T15:15:51.137 INFO:tasks.workunit.client.0.smithi046.stderr:246242: waiting osd.0 seq 21474836542 2024-09-18T15:15:51.138 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:15:51.138 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836541 -lt 21474836542 2024-09-18T15:15:51.138 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:15:51.138 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:15:51.138 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:15:51.138 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836542 -lt 21474836542 2024-09-18T15:15:51.138 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:15:51.138 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-326417514499 2024-09-18T15:15:51.138 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:15:51.138 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:15:51.138 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-326417514499 2024-09-18T15:15:51.138 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:15:51.465 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.nt.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-339302416386 2024-09-18T15:15:51.465 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:15:51.466 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:15:51.466 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-339302416386 2024-09-18T15:15:51.466 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:15:51.466 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=339302416386 2024-09-18T15:15:51.466 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 339302416386' 2024-09-18T15:15:51.466 INFO:tasks.workunit.client.0.smithi046.stderr:246238: waiting osd.2 seq 339302416386 2024-09-18T15:15:51.466 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:15:51.466 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 339302416387 -lt 339302416386 2024-09-18T15:15:51.466 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:15:51.466 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-274877906956 2024-09-18T15:15:51.466 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:15:51.466 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:15:51.466 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-274877906956 2024-09-18T15:15:51.467 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:15:51.467 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=274877906956 2024-09-18T15:15:51.467 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 274877906956' 2024-09-18T15:15:51.467 INFO:tasks.workunit.client.0.smithi046.stderr:246238: waiting osd.3 seq 274877906956 2024-09-18T15:15:51.467 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:15:51.467 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 274877906957 -lt 274877906956 2024-09-18T15:15:51.467 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:15:51.467 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:15:51.467 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:15:51.467 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:15:51.467 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:15:51.467 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:15:51.468 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:15:51.468 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:15:51.468 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:15:51.468 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:15:51.468 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:15:51.468 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:15:51.468 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:15:51.890 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:15:51.891 INFO:tasks.workunit.client.0.smithi046.stderr:246238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:15:51.891 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:15:51.891 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:15:51.891 INFO:tasks.workunit.client.0.smithi046.stderr:246238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:15:51.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:15:51.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:15:51.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 246239 2024-09-18T15:15:52.521 INFO:tasks.workunit.client.0.smithi046.stderr:0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=326417514499 2024-09-18T15:15:52.521 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 326417514499' 2024-09-18T15:15:52.522 INFO:tasks.workunit.client.0.smithi046.stderr:246242: waiting osd.1 seq 326417514499 2024-09-18T15:15:52.522 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:15:52.522 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 326417514499 -lt 326417514499 2024-09-18T15:15:52.522 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:15:52.522 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-339302416387 2024-09-18T15:15:52.522 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:15:52.523 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:15:52.523 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-339302416387 2024-09-18T15:15:52.523 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:15:52.523 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=339302416387 2024-09-18T15:15:52.523 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 339302416387' 2024-09-18T15:15:52.523 INFO:tasks.workunit.client.0.smithi046.stderr:246242: waiting osd.2 seq 339302416387 2024-09-18T15:15:52.523 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:15:52.524 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 339302416387 -lt 339302416387 2024-09-18T15:15:52.524 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:15:52.524 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-274877906957 2024-09-18T15:15:52.524 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:15:52.524 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:15:52.524 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-274877906957 2024-09-18T15:15:52.524 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:15:52.524 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=274877906957 2024-09-18T15:15:52.524 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 274877906957' 2024-09-18T15:15:52.524 INFO:tasks.workunit.client.0.smithi046.stderr:246242: waiting osd.3 seq 274877906957 2024-09-18T15:15:52.524 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:15:52.524 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 274877906958 -lt 274877906957 2024-09-18T15:15:52.524 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:15:52.525 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:15:52.525 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:15:52.525 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:15:52.525 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:15:52.525 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:15:52.525 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:15:52.525 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:15:52.525 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:15:53.229 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:15:53.230 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:15:53.230 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:15:53.230 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:15:53.230 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:15:53.230 INFO:tasks.workunit.client.0.smithi046.stderr:246242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:15:53.230 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:15:53.230 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:15:53.230 INFO:tasks.workunit.client.0.smithi046.stderr:246242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:15:53.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:15:53.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: eval 'pids='\'''\''' 2024-09-18T15:15:53.231 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: pids= 2024-09-18T15:15:53.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: wait_background: return 0 2024-09-18T15:15:53.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: return_code=0 2024-09-18T15:15:53.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-18T15:15:53.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:214: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-18T15:15:53.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-18T15:15:53.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-09-18T15:15:53.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:184: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-18T15:15:53.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local poolname=ecpool 2024-09-18T15:15:53.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local first=3 2024-09-18T15:15:53.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local second=1 2024-09-18T15:15:53.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:192: corrupt_and_repair_two: pids= 2024-09-18T15:15:53.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:15:53.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:15:53.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:15:53.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 249102"' 2024-09-18T15:15:53.286 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 249102' 2024-09-18T15:15:53.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:15:53.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:15:53.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:15:53.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 249104"' 2024-09-18T15:15:53.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:15:53.287 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 249104' 2024-09-18T15:15:53.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: wait_background pids 2024-09-18T15:15:53.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: wait_background: pids=' 249102 249104' 2024-09-18T15:15:53.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/249105: /' 2024-09-18T15:15:53.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: wait_background: return_code=0 2024-09-18T15:15:53.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:15:53.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 249102 2024-09-18T15:15:53.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:15:53.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/249107: /' 2024-09-18T15:15:57.127 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:15:57.127 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:15:57.127 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:15:57.127 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:15:57.127 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:15:57.127 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:15:57.127 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:15:57.127 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:15:57.128 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:15:57.128 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:15:57.128 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:15:57.128 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:15:57.128 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:15:57.128 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:15:57.128 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:15:57.128 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:15:57.128 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:15:57.128 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:15:57.128 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:15:57.128 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:15:57.128 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:15:57.129 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:15:57.129 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-18T15:15:57.129 INFO:tasks.workunit.client.0.smithi046.stderr:249107: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-18T15:15:57.129 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:15:57.129 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:15:57.129 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:15:57.129 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:15:57.129 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:15:57.129 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:15:57.129 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:15:57.129 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:15:57.133 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:15:57.133 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:15:57.133 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:15:57.134 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:15:57.134 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:15:57.134 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:15:57.134 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:15:57.134 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:15:57.134 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:15:57.134 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:15:57.134 INFO:tasks.workunit.client.0.smithi046.stderr:249107: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:15:57.134 INFO:tasks.workunit.client.0.smithi046.stderr:249107: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:15:57.134 INFO:tasks.workunit.client.0.smithi046.stderr:249107: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:15:57.134 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:15:57.134 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:15:57.134 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:15:57.134 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:15:57.135 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:15:57.135 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:15:57.135 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:15:57.135 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:15:57.135 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:15:57.135 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:15:57.135 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:15:57.135 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:15:57.135 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:15:57.135 INFO:tasks.workunit.client.0.smithi046.stderr:249107: start osd.1 2024-09-18T15:15:57.285 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:15:57.285 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:15:57.285 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-18T15:15:57.285 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:15:57.286 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:15:57.286 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:15:57.286 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:15:57.286 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-18T15:15:57.286 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:15:57.286 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-18T15:15:57.286 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:15:57.286 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:15:57.286 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:15:57.286 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:15:57.286 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:15:57.286 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:15:57.286 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:15:57.286 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:15:57.287 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:15:57.287 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-18T15:15:57.287 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:15:57.287 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:15:57.287 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-18T15:15:57.287 INFO:tasks.workunit.client.0.smithi046.stderr:249105: remove 0#2:eb822e21:::SOMETHING:head# 2024-09-18T15:15:57.287 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-18T15:15:57.287 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:15:57.287 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:15:57.287 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T15:15:57.287 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:15:57.287 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:15:57.287 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:15:57.287 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:15:57.291 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:15:57.291 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:15:57.291 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-18T15:15:57.291 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-18T15:15:57.291 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:15:57.291 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:15:57.291 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:15:57.292 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:15:57.292 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:15:57.292 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:15:57.292 INFO:tasks.workunit.client.0.smithi046.stderr:249105: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:15:57.292 INFO:tasks.workunit.client.0.smithi046.stderr:249105: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:15:57.292 INFO:tasks.workunit.client.0.smithi046.stderr:249105: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:15:57.292 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:15:57.292 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:15:57.292 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:15:57.292 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:15:57.292 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:15:57.292 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:15:57.293 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:15:57.293 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:15:57.293 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:15:57.293 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:15:57.293 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:15:57.293 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-18T15:15:57.293 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T15:15:57.293 INFO:tasks.workunit.client.0.smithi046.stderr:249105: start osd.3 2024-09-18T15:15:57.293 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-18T15:16:02.138 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osdops 2024-09-18T15:16:02.138 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:16:02.138 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:16:02.138 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:16:02.138 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:16:02.138 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:16:02.139 INFO:tasks.workunit.client.0.smithi046.stderr:249107: 2024-09-18T15:15:57.154+0000 7f6e8e20c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:02.139 INFO:tasks.workunit.client.0.smithi046.stderr:249107: 2024-09-18T15:15:57.158+0000 7f6e8e20c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:02.139 INFO:tasks.workunit.client.0.smithi046.stderr:249107: 2024-09-18T15:15:57.162+0000 7f6e8e20c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:02.139 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:16:02.139 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:16:02.139 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:16:02.139 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:16:02.139 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:16:02.139 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:02.139 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:16:02.139 INFO:tasks.workunit.client.0.smithi046.stderr:249107: 0 2024-09-18T15:16:02.139 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:02.139 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:02.140 INFO:tasks.workunit.client.0.smithi046.stderr:249107: 2024-09-18T15:15:57.466+0000 7f6e8e20c540 -1 Falling back to public interface 2024-09-18T15:16:02.140 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:02.140 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:02.140 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:02.140 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:16:02.140 INFO:tasks.workunit.client.0.smithi046.stderr:249107: 1 2024-09-18T15:16:02.140 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:02.140 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:02.140 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:02.140 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:02.140 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:02.140 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:16:02.140 INFO:tasks.workunit.client.0.smithi046.stderr:249107: 2 2024-09-18T15:16:02.140 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:02.141 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:02.141 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:02.141 INFO:tasks.workunit.client.0.smithi046.stderr:249107: 2024-09-18T15:16:00.486+0000 7f6e8e20c540 -1 osd.1 81 log_to_monitors true 2024-09-18T15:16:02.141 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:02.141 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:02.141 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:16:02.141 INFO:tasks.workunit.client.0.smithi046.stderr:249107: 3 2024-09-18T15:16:02.141 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:02.141 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:02.141 INFO:tasks.workunit.client.0.smithi046.stderr:249107: osd.1 up in weight 1 up_from 86 up_thru 86 down_at 82 l-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:16:02.142 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T15:16:02.142 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:16:02.142 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:16:02.142 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:16:02.142 INFO:tasks.workunit.client.0.smithi046.stderr:249105: 2024-09-18T15:15:57.310+0000 7fe470859540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:02.142 INFO:tasks.workunit.client.0.smithi046.stderr:249105: 2024-09-18T15:15:57.314+0000 7fe470859540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:02.142 INFO:tasks.workunit.client.0.smithi046.stderr:249105: 2024-09-18T15:15:57.322+0000 7fe470859540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:02.142 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T15:16:02.142 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:16:02.142 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:16:02.142 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:16:02.142 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:16:02.143 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:02.143 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:16:02.143 INFO:tasks.workunit.client.0.smithi046.stderr:249105: 0 2024-09-18T15:16:02.143 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:02.143 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:16:02.143 INFO:tasks.workunit.client.0.smithi046.stderr:249105: 2024-09-18T15:15:57.878+0000 7fe470859540 -1 Falling back to public interface 2024-09-18T15:16:02.143 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:02.143 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:02.143 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:02.143 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:16:02.143 INFO:tasks.workunit.client.0.smithi046.stderr:249105: 1 2024-09-18T15:16:02.143 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:02.143 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:16:02.143 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:02.144 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:02.144 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:02.144 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:16:02.144 INFO:tasks.workunit.client.0.smithi046.stderr:249105: 2 2024-09-18T15:16:02.144 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:02.144 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:16:02.144 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:02.144 INFO:tasks.workunit.client.0.smithi046.stderr:249105: 2024-09-18T15:16:01.450+0000 7fe470859540 -1 osd.3 81 log_to_monitors true 2024-09-18T15:16:02.144 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:02.144 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:02.144 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:16:02.144 INFO:tasks.workunit.client.0.smithi046.stderr:249105: 3 2024-09-18T15:16:02.144 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:02.144 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:16:03.316 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-heast_clean_interval [76,81) [v2:127.0.0.1:6810/249912,v1:127.0.0.1:6811/249912] [v2:127.0.0.1:6812/249912,v1:127.0.0.1:6813/249912] exists,up b59f73a0-2db8-4a9b-ae5e-0fcbadbde7ca 2024-09-18T15:16:03.316 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:16:03.316 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:16:03.316 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:16:03.317 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:16:03.317 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:16:03.317 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:16:03.317 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:16:03.317 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:16:03.317 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:16:03.317 INFO:tasks.workunit.client.0.smithi046.stderr:249107: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:16:03.317 INFO:tasks.workunit.client.0.smithi046.stderr:249107: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:16:03.317 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:16:03.317 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:16:03.317 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:16:03.317 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:16:03.317 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:16:03.317 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:16:03.318 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:16:03.318 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:16:03.318 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:16:03.318 INFO:tasks.workunit.client.0.smithi046.stderr:249107: 1 2024-09-18T15:16:03.318 INFO:tasks.workunit.client.0.smithi046.stderr:249107: 2 2024-09-18T15:16:03.318 INFO:tasks.workunit.client.0.smithi046.stderr:249107: 3' 2024-09-18T15:16:03.318 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:16:03.318 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:16:03.318 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:16:03.318 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836546 2024-09-18T15:16:03.318 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836546 2024-09-18T15:16:03.318 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836546' 2024-09-18T15:16:03.318 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:16:03.318 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:16:03.319 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=369367187458 2024-09-18T15:16:03.319 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 369367187458 2024-09-18T15:16:03.319 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836546 1-369367187458' 2024-09-18T15:16:03.319 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:16:03.319 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:16:04.445 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /homlpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:04.445 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:04.445 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:04.445 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:16:04.445 INFO:tasks.workunit.client.0.smithi046.stderr:249105: 4 2024-09-18T15:16:04.446 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:04.446 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:16:04.446 INFO:tasks.workunit.client.0.smithi046.stderr:249105: osd.3 up in weight 1 up_from 88 up_thru 88 down_at 82 last_clean_interval [64,81) [v2:127.0.0.1:6826/249957,v1:127.0.0.1:6827/249957] [v2:127.0.0.1:6828/249957,v1:127.0.0.1:6829/249957] exists,up db129174-c3ee-42d9-b16d-41a610ac05c9 2024-09-18T15:16:04.446 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:16:04.446 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:16:04.446 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:16:04.446 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:16:04.446 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:16:04.446 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:16:04.446 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:16:04.446 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:16:04.446 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:16:04.446 INFO:tasks.workunit.client.0.smithi046.stderr:249105: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:16:04.446 INFO:tasks.workunit.client.0.smithi046.stderr:249105: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:16:04.447 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:16:04.447 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:16:04.447 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:16:04.447 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:16:04.447 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:16:04.447 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:16:04.447 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:16:04.447 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:16:04.447 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:16:04.447 INFO:tasks.workunit.client.0.smithi046.stderr:249105: 1 2024-09-18T15:16:04.447 INFO:tasks.workunit.client.0.smithi046.stderr:249105: 2 2024-09-18T15:16:04.447 INFO:tasks.workunit.client.0.smithi046.stderr:249105: 3' 2024-09-18T15:16:04.447 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:16:04.447 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:16:04.448 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:16:04.448 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836547 2024-09-18T15:16:04.448 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836547 2024-09-18T15:16:04.448 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836547' 2024-09-18T15:16:04.448 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:16:05.589 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tee/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=339302416391 2024-09-18T15:16:05.590 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 339302416391 2024-09-18T15:16:05.590 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836546 1-369367187458 2-339302416391' 2024-09-18T15:16:05.590 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:16:05.590 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:16:05.590 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=377957122050 2024-09-18T15:16:05.590 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 377957122050 2024-09-18T15:16:05.590 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836546 1-369367187458 2-339302416391 3-377957122050' 2024-09-18T15:16:05.590 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:16:05.590 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836546 2024-09-18T15:16:05.590 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:16:05.591 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:16:05.591 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836546 2024-09-18T15:16:05.591 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:16:05.591 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836546 2024-09-18T15:16:05.591 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836546' 2024-09-18T15:16:05.591 INFO:tasks.workunit.client.0.smithi046.stderr:249107: waiting osd.0 seq 21474836546 2024-09-18T15:16:05.591 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:16:05.591 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836545 -lt 21474836546 2024-09-18T15:16:05.591 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:16:05.591 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:16:05.591 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:16:05.591 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836546 -lt 21474836546 2024-09-18T15:16:05.591 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:16:05.592 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-369367187458 2024-09-18T15:16:05.592 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:16:05.592 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:16:05.592 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-369367187458 2024-09-18T15:16:05.592 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:16:05.592 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=369367187458 2024-09-18T15:16:05.592 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 369367187458' 2024-09-18T15:16:05.592 INFO:tasks.workunit.client.0.smithi046.stderr:249107: waiting osd.1 seq 369367187458 2024-09-18T15:16:05.592 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:16:05.592 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 369367187458 -lt 369367187458 2024-09-18T15:16:05.592 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:16:06.522 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.clienll osd.1 flush_pg_stats 2024-09-18T15:16:06.522 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=369367187459 2024-09-18T15:16:06.522 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 369367187459 2024-09-18T15:16:06.522 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836547 1-369367187459' 2024-09-18T15:16:06.522 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:16:06.522 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:16:06.522 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=339302416392 2024-09-18T15:16:06.522 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 339302416392 2024-09-18T15:16:06.522 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836547 1-369367187459 2-339302416392' 2024-09-18T15:16:06.522 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:16:06.523 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:16:06.523 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=377957122051 2024-09-18T15:16:06.523 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 377957122051 2024-09-18T15:16:06.523 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836547 1-369367187459 2-339302416392 3-377957122051' 2024-09-18T15:16:06.523 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:16:06.523 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836547 2024-09-18T15:16:06.523 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:16:06.523 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:16:06.523 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836547 2024-09-18T15:16:06.523 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:16:06.523 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836547 2024-09-18T15:16:06.523 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836547' 2024-09-18T15:16:06.523 INFO:tasks.workunit.client.0.smithi046.stderr:249105: waiting osd.0 seq 21474836547 2024-09-18T15:16:06.523 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:16:06.524 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836546 -lt 21474836547 2024-09-18T15:16:06.524 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:16:06.524 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:16:06.524 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:16:06.524 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836548 -lt 21474836547 2024-09-18T15:16:06.524 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:16:06.524 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-369367187459 2024-09-18T15:16:06.524 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:16:06.524 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:16:06.524 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-369367187459 2024-09-18T15:16:06.524 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:16:07.049 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standt.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-339302416391 2024-09-18T15:16:07.050 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:16:07.050 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:16:07.050 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-339302416391 2024-09-18T15:16:07.050 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:16:07.050 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=339302416391 2024-09-18T15:16:07.050 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 339302416391' 2024-09-18T15:16:07.050 INFO:tasks.workunit.client.0.smithi046.stderr:249107: waiting osd.2 seq 339302416391 2024-09-18T15:16:07.050 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:16:07.050 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 339302416392 -lt 339302416391 2024-09-18T15:16:07.050 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:16:07.051 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-377957122050 2024-09-18T15:16:07.051 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:16:07.051 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:16:07.051 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-377957122050 2024-09-18T15:16:07.051 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:16:07.051 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=377957122050 2024-09-18T15:16:07.051 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 377957122050' 2024-09-18T15:16:07.051 INFO:tasks.workunit.client.0.smithi046.stderr:249107: waiting osd.3 seq 377957122050 2024-09-18T15:16:07.051 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:16:07.051 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 377957122051 -lt 377957122050 2024-09-18T15:16:07.051 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:16:07.052 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:16:07.052 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:16:07.052 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:16:07.052 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:16:07.052 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:16:07.052 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:16:07.052 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:16:07.052 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:16:07.052 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:16:07.052 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:16:07.053 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:16:07.053 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:16:07.450 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:16:07.451 INFO:tasks.workunit.client.0.smithi046.stderr:249107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:16:07.451 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:16:07.451 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:16:07.451 INFO:tasks.workunit.client.0.smithi046.stderr:249107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:16:07.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:16:07.907 INFO:tasks.workunit.client.0.smithi046.stderr:alone/ceph-helpers.sh:2183: flush_pg_stats: seq=369367187459 2024-09-18T15:16:07.907 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 369367187459' 2024-09-18T15:16:07.908 INFO:tasks.workunit.client.0.smithi046.stderr:249105: waiting osd.1 seq 369367187459 2024-09-18T15:16:07.908 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:16:07.908 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 369367187459 -lt 369367187459 2024-09-18T15:16:07.908 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:16:07.908 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:16:07.909 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-339302416392 2024-09-18T15:16:07.909 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:16:07.909 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-339302416392 2024-09-18T15:16:07.909 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:16:07.909 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=339302416392 2024-09-18T15:16:07.910 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 339302416392' 2024-09-18T15:16:07.910 INFO:tasks.workunit.client.0.smithi046.stderr:249105: waiting osd.2 seq 339302416392 2024-09-18T15:16:07.910 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:16:07.910 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 339302416392 -lt 339302416392 2024-09-18T15:16:07.910 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:16:07.910 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-377957122051 2024-09-18T15:16:07.910 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:16:07.910 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:16:07.910 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:16:07.911 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-377957122051 2024-09-18T15:16:07.911 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=377957122051 2024-09-18T15:16:07.911 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 377957122051' 2024-09-18T15:16:07.911 INFO:tasks.workunit.client.0.smithi046.stderr:249105: waiting osd.3 seq 377957122051 2024-09-18T15:16:07.911 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:16:07.911 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 377957122051 -lt 377957122051 2024-09-18T15:16:07.911 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:16:07.911 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:16:07.912 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:16:07.912 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:16:07.912 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:16:07.912 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:16:07.912 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:16:07.912 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:16:07.912 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:16:08.580 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:16:08.580 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:16:08.580 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:16:08.580 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:16:08.580 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:16:08.581 INFO:tasks.workunit.client.0.smithi046.stderr:249105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:16:08.581 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:16:08.581 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:16:08.581 INFO:tasks.workunit.client.0.smithi046.stderr:249105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:16:08.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:16:08.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:16:08.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 249104 2024-09-18T15:16:08.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: eval 'pids='\'''\''' 2024-09-18T15:16:08.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: pids= 2024-09-18T15:16:08.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: wait_background: return 0 2024-09-18T15:16:08.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: return_code=0 2024-09-18T15:16:08.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-18T15:16:08.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-18T15:16:08.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-09-18T15:16:08.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T15:16:08.583 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:16:08.583 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:16:08.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: local pg=2.0 2024-09-18T15:16:08.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: repair 2.0 2024-09-18T15:16:08.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-09-18T15:16:08.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-09-18T15:16:08.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:16:08.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:16:08.890 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:16:08.890 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:16:09.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T15:15:34.131205+0000 2024-09-18T15:16:09.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-09-18T15:16:09.544 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-18T15:16:09.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-09-18T15:15:34.131205+0000 2024-09-18T15:16:09.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=2.0 2024-09-18T15:16:09.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:15:34.131205+0000 2024-09-18T15:16:09.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:16:09.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:16:09.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:16:09.554 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:16:09.554 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:16:09.554 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:16:09.554 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:16:09.554 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:16:09.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:15:34.131205+0000 '>' 2024-09-18T15:15:34.131205+0000 2024-09-18T15:16:09.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:16:10.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:16:10.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:16:10.856 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:16:10.856 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:16:10.856 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:16:10.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:16:10.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:16:11.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:15:34.131205+0000 '>' 2024-09-18T15:15:34.131205+0000 2024-09-18T15:16:11.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:16:12.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:16:12.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:16:12.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:16:12.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:16:12.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:16:12.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:16:12.172 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:16:12.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:15:34.131205+0000 '>' 2024-09-18T15:15:34.131205+0000 2024-09-18T15:16:12.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:16:13.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:16:13.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:16:13.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:16:13.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:16:13.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:16:13.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:16:13.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:16:13.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:15:34.131205+0000 '>' 2024-09-18T15:15:34.131205+0000 2024-09-18T15:16:13.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:16:14.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:16:14.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:16:14.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:16:14.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:16:14.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:16:14.824 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:16:14.824 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:16:15.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:15:34.131205+0000 '>' 2024-09-18T15:15:34.131205+0000 2024-09-18T15:16:15.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:16:16.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:16:16.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:16:16.143 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:16:16.143 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:16:16.143 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:16:16.144 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:16:16.144 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:16:16.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:15:34.131205+0000 '>' 2024-09-18T15:15:34.131205+0000 2024-09-18T15:16:16.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:16:17.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:16:17.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:16:17.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:16:17.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:16:17.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:16:17.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:16:17.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:16:17.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:16:12.410341+0000 '>' 2024-09-18T15:15:34.131205+0000 2024-09-18T15:16:17.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:16:17.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:207: corrupt_and_repair_two: pids= 2024-09-18T15:16:17.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:16:17.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:16:17.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:16:17.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 252200"' 2024-09-18T15:16:17.804 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 252200' 2024-09-18T15:16:17.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:16:17.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:16:17.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:16:17.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:16:17.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 252202"' 2024-09-18T15:16:17.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/252203: /' 2024-09-18T15:16:17.806 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 252202' 2024-09-18T15:16:17.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: wait_background pids 2024-09-18T15:16:17.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: wait_background: pids=' 252200 252202' 2024-09-18T15:16:17.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: wait_background: return_code=0 2024-09-18T15:16:17.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:16:17.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 252200 2024-09-18T15:16:17.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:16:17.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/252205: /' 2024-09-18T15:16:21.126 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:16:21.126 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:16:21.127 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-18T15:16:21.127 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:16:21.127 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:16:21.127 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:16:21.127 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:16:21.127 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-18T15:16:21.127 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:16:21.127 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-18T15:16:21.127 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:16:21.127 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:16:21.127 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:16:21.127 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:16:21.127 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:16:21.128 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:16:21.128 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:16:21.128 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:16:21.128 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:16:21.128 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-18T15:16:21.128 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:16:21.128 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:16:21.128 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-18T15:16:21.128 INFO:tasks.workunit.client.0.smithi046.stderr:252203: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-18T15:16:21.128 INFO:tasks.workunit.client.0.smithi046.stderr:252203: _ 2024-09-18T15:16:21.128 INFO:tasks.workunit.client.0.smithi046.stderr:252203: hinfo_key 2024-09-18T15:16:21.128 INFO:tasks.workunit.client.0.smithi046.stderr:252203: snapset 2024-09-18T15:16:21.128 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-18T15:16:21.129 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:16:21.129 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:16:21.129 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T15:16:21.129 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:16:21.129 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:16:21.129 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:16:21.133 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:16:21.133 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:16:21.133 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:16:21.133 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-18T15:16:21.133 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-18T15:16:21.133 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:16:21.133 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:16:21.133 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:16:21.134 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:16:21.134 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:16:21.134 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:16:21.134 INFO:tasks.workunit.client.0.smithi046.stderr:252203: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:16:21.134 INFO:tasks.workunit.client.0.smithi046.stderr:252203: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:16:21.134 INFO:tasks.workunit.client.0.smithi046.stderr:252203: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:16:21.134 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:16:21.134 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:16:21.134 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:16:21.134 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:16:21.134 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:16:21.134 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:16:21.134 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:16:21.135 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:16:21.135 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:16:21.135 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:16:21.135 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:16:21.135 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-18T15:16:21.135 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T15:16:21.135 INFO:tasks.workunit.client.0.smithi046.stderr:252203: start osd.3 2024-09-18T15:16:21.497 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namesp252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:16:21.498 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:16:21.498 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:16:21.498 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:16:21.498 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:16:21.498 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:16:21.498 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:16:21.498 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:16:21.498 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:16:21.498 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:16:21.498 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:16:21.498 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:16:21.498 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:16:21.499 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:16:21.499 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:16:21.499 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:16:21.499 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:16:21.499 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:16:21.499 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:16:21.499 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:16:21.499 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:16:21.499 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:16:21.499 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-18T15:16:21.499 INFO:tasks.workunit.client.0.smithi046.stderr:252205: _ 2024-09-18T15:16:21.499 INFO:tasks.workunit.client.0.smithi046.stderr:252205: hinfo_key 2024-09-18T15:16:21.499 INFO:tasks.workunit.client.0.smithi046.stderr:252205: snapset 2024-09-18T15:16:21.499 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:16:21.499 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:16:21.500 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:16:21.500 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:16:21.500 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:16:21.500 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:16:21.500 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:16:21.500 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:16:21.503 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:16:21.503 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:16:21.503 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:16:21.503 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:16:21.503 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:16:21.503 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:16:21.503 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:16:21.503 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:16:21.503 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:16:21.504 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:16:21.504 INFO:tasks.workunit.client.0.smithi046.stderr:252205: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:16:21.504 INFO:tasks.workunit.client.0.smithi046.stderr:252205: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:16:21.504 INFO:tasks.workunit.client.0.smithi046.stderr:252205: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:16:21.504 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:16:21.504 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:16:21.504 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:16:21.504 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:16:21.504 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:16:21.504 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:16:21.504 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:16:21.504 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:16:21.504 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:16:21.504 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:16:21.504 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:16:21.505 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:16:21.505 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:16:21.505 INFO:tasks.workunit.client.0.smithi046.stderr:252205: start osd.1 2024-09-18T15:16:25.436 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=highace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:16:25.437 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-18T15:16:25.437 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T15:16:25.437 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:16:25.437 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:16:25.437 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:16:25.437 INFO:tasks.workunit.client.0.smithi046.stderr:252203: 2024-09-18T15:16:21.154+0000 7f6e39934540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:25.437 INFO:tasks.workunit.client.0.smithi046.stderr:252203: 2024-09-18T15:16:21.158+0000 7f6e39934540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:25.437 INFO:tasks.workunit.client.0.smithi046.stderr:252203: 2024-09-18T15:16:21.162+0000 7f6e39934540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:25.437 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T15:16:25.438 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:16:25.438 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:16:25.438 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:16:25.438 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:16:25.438 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:25.438 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:16:25.438 INFO:tasks.workunit.client.0.smithi046.stderr:252203: 0 2024-09-18T15:16:25.438 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:16:25.438 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:25.438 INFO:tasks.workunit.client.0.smithi046.stderr:252203: 2024-09-18T15:16:21.714+0000 7f6e39934540 -1 Falling back to public interface 2024-09-18T15:16:25.438 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:25.438 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:25.438 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:25.438 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:16:25.439 INFO:tasks.workunit.client.0.smithi046.stderr:252203: 1 2024-09-18T15:16:25.439 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:25.439 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:16:25.439 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:25.439 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:25.439 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:25.439 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:16:25.439 INFO:tasks.workunit.client.0.smithi046.stderr:252203: 2 2024-09-18T15:16:25.439 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:25.439 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:16:25.439 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:25.439 INFO:tasks.workunit.client.0.smithi046.stderr:252203: 2024-09-18T15:16:24.490+0000 7f6e39934540 -1 osd.3 90 log_to_monitors true 2024-09-18T15:16:25.440 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:25.440 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:25.440 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:16:25.440 INFO:tasks.workunit.client.0.smithi046.stderr:252203: 3 2024-09-18T15:16:25.440 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:26.441 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-he_recovery_ops 2024-09-18T15:16:26.441 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:16:26.442 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:16:26.442 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:16:26.442 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:16:26.442 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:16:26.442 INFO:tasks.workunit.client.0.smithi046.stderr:252205: 2024-09-18T15:16:21.522+0000 7fd26ae71540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:26.442 INFO:tasks.workunit.client.0.smithi046.stderr:252205: 2024-09-18T15:16:21.526+0000 7fd26ae71540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:26.442 INFO:tasks.workunit.client.0.smithi046.stderr:252205: 2024-09-18T15:16:21.530+0000 7fd26ae71540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:26.442 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:16:26.442 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:16:26.442 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:16:26.442 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:16:26.442 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:16:26.442 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:26.443 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:16:26.443 INFO:tasks.workunit.client.0.smithi046.stderr:252205: 0 2024-09-18T15:16:26.443 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:26.443 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:26.443 INFO:tasks.workunit.client.0.smithi046.stderr:252205: 2024-09-18T15:16:22.094+0000 7fd26ae71540 -1 Falling back to public interface 2024-09-18T15:16:26.443 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:26.443 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:26.443 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:26.443 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:16:26.443 INFO:tasks.workunit.client.0.smithi046.stderr:252205: 1 2024-09-18T15:16:26.443 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:26.443 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:26.443 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:26.444 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:26.444 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:26.444 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:16:26.444 INFO:tasks.workunit.client.0.smithi046.stderr:252205: 2 2024-09-18T15:16:26.444 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:26.444 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:26.444 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:26.444 INFO:tasks.workunit.client.0.smithi046.stderr:252205: 2024-09-18T15:16:25.554+0000 7fd26ae71540 -1 osd.1 90 log_to_monitors true 2024-09-18T15:16:26.444 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:26.444 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:26.445 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:16:26.445 INFO:tasks.workunit.client.0.smithi046.stderr:252205: 3 2024-09-18T15:16:26.445 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:26.445 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:26.849 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalolpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:16:26.849 INFO:tasks.workunit.client.0.smithi046.stderr:252203: osd.3 up in weight 1 up_from 95 up_thru 95 down_at 91 last_clean_interval [88,90) [v2:127.0.0.1:6810/253010,v1:127.0.0.1:6811/253010] [v2:127.0.0.1:6812/253010,v1:127.0.0.1:6813/253010] exists,up db129174-c3ee-42d9-b16d-41a610ac05c9 2024-09-18T15:16:26.849 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:16:26.849 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:16:26.849 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:16:26.850 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:16:26.850 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:16:26.850 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:16:26.850 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:16:26.850 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:16:26.850 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:16:26.850 INFO:tasks.workunit.client.0.smithi046.stderr:252203: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:16:26.850 INFO:tasks.workunit.client.0.smithi046.stderr:252203: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:16:26.850 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:16:26.850 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:16:26.850 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:16:26.850 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:16:26.851 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:16:26.851 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:16:26.851 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:16:26.851 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:16:26.851 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:16:26.851 INFO:tasks.workunit.client.0.smithi046.stderr:252203: 1 2024-09-18T15:16:26.851 INFO:tasks.workunit.client.0.smithi046.stderr:252203: 2 2024-09-18T15:16:26.851 INFO:tasks.workunit.client.0.smithi046.stderr:252203: 3' 2024-09-18T15:16:26.851 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:16:26.851 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:16:26.851 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:16:26.851 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836553 2024-09-18T15:16:26.851 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836553 2024-09-18T15:16:26.852 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836553' 2024-09-18T15:16:26.852 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:16:26.852 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:16:26.852 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=2 2024-09-18T15:16:26.852 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 2 2024-09-18T15:16:26.852 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836553 1-2' 2024-09-18T15:16:26.852 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:16:28.556 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/cephne/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:28.556 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:28.556 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:28.556 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:16:28.556 INFO:tasks.workunit.client.0.smithi046.stderr:252205: 4 2024-09-18T15:16:28.556 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:28.556 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:28.556 INFO:tasks.workunit.client.0.smithi046.stderr:252205: osd.1 up in weight 1 up_from 98 up_thru 98 down_at 91 last_clean_interval [86,90) [v2:127.0.0.1:6826/253061,v1:127.0.0.1:6827/253061] [v2:127.0.0.1:6828/253061,v1:127.0.0.1:6829/253061] exists,up b59f73a0-2db8-4a9b-ae5e-0fcbadbde7ca 2024-09-18T15:16:28.556 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:16:28.556 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:16:28.556 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:16:28.557 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:16:28.557 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:16:28.557 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:16:28.557 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:16:28.557 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:16:28.557 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:16:28.557 INFO:tasks.workunit.client.0.smithi046.stderr:252205: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:16:28.557 INFO:tasks.workunit.client.0.smithi046.stderr:252205: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:16:28.557 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:16:28.557 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:16:28.557 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:16:28.557 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:16:28.558 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:16:28.558 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:16:28.558 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:16:28.558 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:16:28.558 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:16:28.558 INFO:tasks.workunit.client.0.smithi046.stderr:252205: 1 2024-09-18T15:16:28.558 INFO:tasks.workunit.client.0.smithi046.stderr:252205: 2 2024-09-18T15:16:28.558 INFO:tasks.workunit.client.0.smithi046.stderr:252205: 3' 2024-09-18T15:16:28.558 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:16:28.558 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:16:28.558 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:16:28.558 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836554 2024-09-18T15:16:28.558 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836554 2024-09-18T15:16:28.558 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836554' 2024-09-18T15:16:28.559 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:16:29.366 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:16:29.366 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=339302416398 2024-09-18T15:16:29.366 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 339302416398 2024-09-18T15:16:29.366 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836553 1-2 2-339302416398' 2024-09-18T15:16:29.367 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:16:29.367 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:16:29.367 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=408021893122 2024-09-18T15:16:29.367 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 408021893122 2024-09-18T15:16:29.367 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836553 1-2 2-339302416398 3-408021893122' 2024-09-18T15:16:29.367 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:16:29.367 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836553 2024-09-18T15:16:29.367 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:16:29.367 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:16:29.367 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836553 2024-09-18T15:16:29.367 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:16:29.367 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836553 2024-09-18T15:16:29.367 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836553' 2024-09-18T15:16:29.368 INFO:tasks.workunit.client.0.smithi046.stderr:252203: waiting osd.0 seq 21474836553 2024-09-18T15:16:29.368 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:16:29.368 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836552 -lt 21474836553 2024-09-18T15:16:29.368 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:16:29.368 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:16:29.368 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:16:29.368 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836553 -lt 21474836553 2024-09-18T15:16:29.368 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:16:29.368 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-2 2024-09-18T15:16:29.368 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:16:29.368 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:16:29.368 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-2 2024-09-18T15:16:29.368 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:16:29.368 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=2 2024-09-18T15:16:29.369 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 2' 2024-09-18T15:16:29.369 INFO:tasks.workunit.client.0.smithi046.stderr:252203: waiting osd.1 seq 2 2024-09-18T15:16:29.369 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:16:29.369 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 2 -lt 2 2024-09-18T15:16:29.369 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:16:30.715 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/cep: ceph tell osd.1 flush_pg_stats 2024-09-18T15:16:30.715 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=420906795011 2024-09-18T15:16:30.715 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 420906795011 2024-09-18T15:16:30.716 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836554 1-420906795011' 2024-09-18T15:16:30.716 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:16:30.716 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:16:30.716 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=339302416399 2024-09-18T15:16:30.716 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 339302416399 2024-09-18T15:16:30.716 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836554 1-420906795011 2-339302416399' 2024-09-18T15:16:30.716 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:16:30.717 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:16:30.717 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=408021893123 2024-09-18T15:16:30.717 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 408021893123 2024-09-18T15:16:30.717 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836554 1-420906795011 2-339302416399 3-408021893123' 2024-09-18T15:16:30.717 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:16:30.717 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836554 2024-09-18T15:16:30.717 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:16:30.718 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:16:30.718 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836554 2024-09-18T15:16:30.718 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:16:30.718 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836554 2024-09-18T15:16:30.718 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836554' 2024-09-18T15:16:30.718 INFO:tasks.workunit.client.0.smithi046.stderr:252205: waiting osd.0 seq 21474836554 2024-09-18T15:16:30.718 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:16:30.719 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836553 -lt 21474836554 2024-09-18T15:16:30.719 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:16:30.719 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:16:30.719 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:16:30.719 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836554 -lt 21474836554 2024-09-18T15:16:30.719 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:16:30.719 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-420906795011 2024-09-18T15:16:30.719 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:16:30.720 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:16:30.720 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-420906795011 2024-09-18T15:16:30.720 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:16:30.724 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.h-helpers.sh:2182: flush_pg_stats: echo 2-339302416398 2024-09-18T15:16:30.725 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:16:30.725 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:16:30.725 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:16:30.725 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-339302416398 2024-09-18T15:16:30.725 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=339302416398 2024-09-18T15:16:30.725 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 339302416398' 2024-09-18T15:16:30.725 INFO:tasks.workunit.client.0.smithi046.stderr:252203: waiting osd.2 seq 339302416398 2024-09-18T15:16:30.725 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:16:30.725 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 339302416399 -lt 339302416398 2024-09-18T15:16:30.725 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:16:30.725 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-408021893122 2024-09-18T15:16:30.726 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:16:30.726 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:16:30.726 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-408021893122 2024-09-18T15:16:30.726 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:16:30.726 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=408021893122 2024-09-18T15:16:30.726 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 408021893122' 2024-09-18T15:16:30.726 INFO:tasks.workunit.client.0.smithi046.stderr:252203: waiting osd.3 seq 408021893122 2024-09-18T15:16:30.726 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:16:30.726 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 408021893123 -lt 408021893122 2024-09-18T15:16:30.726 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:16:30.726 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:16:30.726 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:16:30.726 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:16:30.726 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:16:30.727 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:16:30.727 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:16:30.727 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:16:30.727 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:16:30.727 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:16:30.727 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:16:30.727 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:16:30.727 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:16:31.139 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:16:31.139 INFO:tasks.workunit.client.0.smithi046.stderr:252203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:16:31.139 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:16:31.139 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:16:31.139 INFO:tasks.workunit.client.0.smithi046.stderr:252203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:16:31.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:16:31.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:16:31.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 252202 2024-09-18T15:16:32.083 INFO:tasks.workunit.client.0.smithi046.stderr:0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=420906795011 2024-09-18T15:16:32.083 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 420906795011' 2024-09-18T15:16:32.083 INFO:tasks.workunit.client.0.smithi046.stderr:252205: waiting osd.1 seq 420906795011 2024-09-18T15:16:32.083 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:16:32.083 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 420906795011 -lt 420906795011 2024-09-18T15:16:32.083 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:16:32.083 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-339302416399 2024-09-18T15:16:32.083 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:16:32.083 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:16:32.084 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-339302416399 2024-09-18T15:16:32.084 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:16:32.084 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=339302416399 2024-09-18T15:16:32.084 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 339302416399' 2024-09-18T15:16:32.084 INFO:tasks.workunit.client.0.smithi046.stderr:252205: waiting osd.2 seq 339302416399 2024-09-18T15:16:32.084 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:16:32.084 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 339302416399 -lt 339302416399 2024-09-18T15:16:32.084 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:16:32.084 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-408021893123 2024-09-18T15:16:32.084 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:16:32.084 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:16:32.084 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-408021893123 2024-09-18T15:16:32.085 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:16:32.085 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=408021893123 2024-09-18T15:16:32.085 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 408021893123' 2024-09-18T15:16:32.085 INFO:tasks.workunit.client.0.smithi046.stderr:252205: waiting osd.3 seq 408021893123 2024-09-18T15:16:32.085 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:16:32.085 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 408021893124 -lt 408021893123 2024-09-18T15:16:32.085 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:16:32.085 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:16:32.085 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:16:32.085 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:16:32.085 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:16:32.085 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:16:32.085 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:16:32.086 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:16:32.086 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:16:32.789 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:16:32.789 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:16:32.789 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:16:32.789 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:16:32.789 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:16:32.789 INFO:tasks.workunit.client.0.smithi046.stderr:252205: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:16:32.789 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:16:32.789 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:16:32.789 INFO:tasks.workunit.client.0.smithi046.stderr:252205: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:16:32.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:16:32.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: eval 'pids='\'''\''' 2024-09-18T15:16:32.790 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: pids= 2024-09-18T15:16:32.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: wait_background: return 0 2024-09-18T15:16:32.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: return_code=0 2024-09-18T15:16:32.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-18T15:16:32.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:214: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-18T15:16:32.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-18T15:16:32.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T15:16:32.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:16:32.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:16:32.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:16:32.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:16:32.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:16:32.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:16:32.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:16:32.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:16:32.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:16:32.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:16:32.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:16:32.963 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:16:32.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:16:32.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:16:32.967 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:16:32.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:16:32.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:16:32.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:16:32.969 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:16:32.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:16:32.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:16:32.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:16:33.007 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:16:33.007 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:16:33.007 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:16:33.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:16:33.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:16:33.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:16:33.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T15:16:33.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T15:16:33.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T15:16:33.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T15:16:33.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:16:33.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:16:33.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:16:33.015 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:16:33.015 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:16:33.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:16:33.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:16:33.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:16:33.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:16:33.016 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:16:33.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:16:33.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:16:33.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:16:33.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:16:33.020 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:16:33.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:16:33.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:16:33.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:16:33.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:16:33.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:16:33.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:16:33.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:16:33.028 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:16:33.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:16:33.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:16:33.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:16:33.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:16:33.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:16:33.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T15:16:33.033 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:16:33.034 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:16:33.034 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:16:33.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T15:16:33.037 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:16:33.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:16:33.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:16:33.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T15:16:33.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_corrupt_and_repair_lrc_appends td/osd-scrub-repair 2024-09-18T15:16:33.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:851: TEST_corrupt_and_repair_lrc_appends: corrupt_and_repair_lrc td/osd-scrub-repair false 2024-09-18T15:16:33.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-09-18T15:16:33.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: local allow_overwrites=false 2024-09-18T15:16:33.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: local poolname=ecpool 2024-09-18T15:16:33.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-09-18T15:16:33.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T15:16:33.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:16:33.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:16:33.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:16:33.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T15:16:33.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-18T15:16:33.430 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:16:33.430 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:16:33.430 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:16:33.431 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:16:33.431 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:16:33.431 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:16:33.431 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:16:33.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T15:16:33.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:16:33.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:16:33.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:16:33.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:16:33.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:16:33.772 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:16:33.772 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:16:33.772 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:16:33.772 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:16:33.772 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:16:33.773 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:16:33.773 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:16:33.773 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:16:33.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:16:33.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T15:16:33.871 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:16:33.871 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:16:33.871 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:16:33.871 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:16:33.871 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:16:33.872 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:16:33.872 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:16:33.872 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:16:33.873 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:16:33.873 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:16:33.873 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:16:33.873 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:16:33.874 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:16:33.874 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T15:16:33.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-09-18T15:16:33.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T15:16:33.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:16:33.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:16:33.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:16:33.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T15:16:33.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:16:34.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:16:34.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:16:34.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:16:34.823 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:16:34.823 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:16:34.823 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:16:34.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:16:34.824 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:16:34.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:16:34.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: seq 0 9 2024-09-18T15:16:34.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-18T15:16:34.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: '[' false = true ']' 2024-09-18T15:16:34.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:840: corrupt_and_repair_lrc: run_osd_filestore td/osd-scrub-repair 0 2024-09-18T15:16:34.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T15:16:34.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T15:16:34.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=0 2024-09-18T15:16:34.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T15:16:34.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:16:34.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:16:34.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:16:34.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:16:34.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:16:34.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:16:34.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:16:34.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T15:16:34.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T15:16:34.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:16:34.868 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T15:16:34.868 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:16:34.868 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:16:34.868 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:16:34.868 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:16:34.868 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:16:34.868 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:16:34.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:16:34.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T15:16:34.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T15:16:34.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T15:16:34.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:16:34.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:16:34.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:16:34.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:16:34.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:16:34.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T15:16:34.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T15:16:34.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:16:34.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T15:16:34.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=98f43c13-4f9d-40be-be0b-274247f4620d 2024-09-18T15:16:34.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd3 98f43c13-4f9d-40be-be0b-274247f4620d' 2024-09-18T15:16:34.872 INFO:tasks.workunit.client.0.smithi046.stdout:add osd3 98f43c13-4f9d-40be-be0b-274247f4620d 2024-09-18T15:16:34.873 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T15:16:34.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQDS7upmK9TFNBAAPxum4eAc1gfC//U/QLHhTw== 2024-09-18T15:16:34.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQDS7upmK9TFNBAAPxum4eAc1gfC//U/QLHhTw=="}' 2024-09-18T15:16:34.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 98f43c13-4f9d-40be-be0b-274247f4620d -i td/osd-scrub-repair/0/new.json 2024-09-18T15:16:35.154 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:16:35.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/0/new.json 2024-09-18T15:16:35.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDS7upmK9TFNBAAPxum4eAc1gfC//U/QLHhTw== --osd-uuid 98f43c13-4f9d-40be-be0b-274247f4620d --osd-objectstore=filestore 2024-09-18T15:16:35.195 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:35.186+0000 7f8671235540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:35.199 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:35.190+0000 7f8671235540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:35.200 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:35.190+0000 7f8671235540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:35.731 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:35.722+0000 7f8671235540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:16:36.202 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:36.194+0000 7f8671235540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:16:36.202 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:36.194+0000 7f8671235540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:16:36.202 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:36.194+0000 7f8671235540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:16:36.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T15:16:36.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T15:16:36.795 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T15:16:36.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd0 key to auth repository 2024-09-18T15:16:36.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:16:36.980 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: key for osd.0 not found in provided keyring 2024-09-18T15:16:36.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.0 2024-09-18T15:16:36.983 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T15:16:36.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:16:36.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T15:16:36.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T15:16:36.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T15:16:37.012 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:37.006+0000 7fa5b5991540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:37.015 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:37.006+0000 7fa5b5991540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:37.016 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:37.006+0000 7fa5b5991540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:37.017 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:37.010+0000 7fa5b5991540 -1 Falling back to public interface 2024-09-18T15:16:37.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 0 2024-09-18T15:16:37.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:16:37.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:16:37.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:16:37.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:16:37.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:37.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:16:37.318 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:16:37.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:37.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:16:37.518 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:37.510+0000 7fa5b5991540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:16:37.518 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:37.510+0000 7fa5b5991540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:16:37.518 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:37.510+0000 7fa5b5991540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:16:37.525 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:37.518+0000 7fa5b5991540 -1 osd.0 0 log_to_monitors true 2024-09-18T15:16:37.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:38.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:38.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:38.571 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:16:38.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:16:38.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:38.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:16:38.770 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/255406,v1:127.0.0.1:6801/255406] [v2:127.0.0.1:6802/255406,v1:127.0.0.1:6803/255406] exists,up 98f43c13-4f9d-40be-be0b-274247f4620d 2024-09-18T15:16:38.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:16:38.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:16:38.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:16:38.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-18T15:16:38.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: '[' false = true ']' 2024-09-18T15:16:38.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:840: corrupt_and_repair_lrc: run_osd_filestore td/osd-scrub-repair 1 2024-09-18T15:16:38.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T15:16:38.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T15:16:38.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=1 2024-09-18T15:16:38.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T15:16:38.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:16:38.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:16:38.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:16:38.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:16:38.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:16:38.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:16:38.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:16:38.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T15:16:38.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T15:16:38.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:16:38.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T15:16:38.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:16:38.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:16:38.775 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:16:38.775 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:16:38.775 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:16:38.776 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:16:38.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:16:38.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T15:16:38.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T15:16:38.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T15:16:38.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:16:38.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:16:38.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:16:38.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:16:38.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:16:38.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T15:16:38.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T15:16:38.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:16:38.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T15:16:38.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=5924be22-2415-4a19-bf4c-9a26d2101e79 2024-09-18T15:16:38.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd3 5924be22-2415-4a19-bf4c-9a26d2101e79' 2024-09-18T15:16:38.779 INFO:tasks.workunit.client.0.smithi046.stdout:add osd3 5924be22-2415-4a19-bf4c-9a26d2101e79 2024-09-18T15:16:38.779 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T15:16:38.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQDW7upmsHNRLxAAVFT+si7wmfSIPaMldqWH/Q== 2024-09-18T15:16:38.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQDW7upmsHNRLxAAVFT+si7wmfSIPaMldqWH/Q=="}' 2024-09-18T15:16:38.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 5924be22-2415-4a19-bf4c-9a26d2101e79 -i td/osd-scrub-repair/1/new.json 2024-09-18T15:16:39.108 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:16:39.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/1/new.json 2024-09-18T15:16:39.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDW7upmsHNRLxAAVFT+si7wmfSIPaMldqWH/Q== --osd-uuid 5924be22-2415-4a19-bf4c-9a26d2101e79 --osd-objectstore=filestore 2024-09-18T15:16:39.150 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:39.142+0000 7f2412cde540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:39.155 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:39.146+0000 7f2412cde540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:39.156 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:39.146+0000 7f2412cde540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:39.647 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:39.638+0000 7f2412cde540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:16:40.159 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:40.150+0000 7f2412cde540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:16:40.178 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:40.170+0000 7f2412cde540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:16:40.178 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:40.170+0000 7f2412cde540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:16:41.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-18T15:16:41.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T15:16:41.349 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T15:16:41.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd1 key to auth repository 2024-09-18T15:16:41.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:16:41.749 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: key for osd.1 not found in provided keyring 2024-09-18T15:16:41.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.1 2024-09-18T15:16:41.753 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T15:16:41.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:16:41.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T15:16:41.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T15:16:41.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T15:16:41.780 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:41.770+0000 7f9537d64540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:41.783 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:41.774+0000 7f9537d64540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:41.785 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:41.778+0000 7f9537d64540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:41.789 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:41.782+0000 7f9537d64540 -1 Falling back to public interface 2024-09-18T15:16:42.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 1 2024-09-18T15:16:42.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:16:42.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:16:42.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:16:42.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:16:42.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:42.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:16:42.106 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:16:42.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:42.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:42.456 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:42.446+0000 7f9537d64540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:16:42.462 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:42.454+0000 7f9537d64540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:16:42.462 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:42.454+0000 7f9537d64540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:16:42.482 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:42.474+0000 7f9537d64540 -1 osd.1 0 log_to_monitors true 2024-09-18T15:16:42.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:43.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:43.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:43.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:16:43.581 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:16:43.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:43.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:43.925 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/255788,v1:127.0.0.1:6811/255788] [v2:127.0.0.1:6812/255788,v1:127.0.0.1:6813/255788] exists,up 5924be22-2415-4a19-bf4c-9a26d2101e79 2024-09-18T15:16:43.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:16:43.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:16:43.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:16:43.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-18T15:16:43.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: '[' false = true ']' 2024-09-18T15:16:43.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:840: corrupt_and_repair_lrc: run_osd_filestore td/osd-scrub-repair 2 2024-09-18T15:16:43.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T15:16:43.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T15:16:43.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=2 2024-09-18T15:16:43.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T15:16:43.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:16:43.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:16:43.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:16:43.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:16:43.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:16:43.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T15:16:43.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T15:16:43.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T15:16:43.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T15:16:43.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:16:43.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T15:16:43.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:16:43.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:16:43.930 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:16:43.930 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:16:43.930 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:16:43.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:16:43.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:16:43.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T15:16:43.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T15:16:43.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T15:16:43.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:16:43.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:16:43.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:16:43.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:16:43.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:16:43.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T15:16:43.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T15:16:43.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/2 2024-09-18T15:16:43.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T15:16:43.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=10ab86bd-bc7e-4208-a5b7-99e6ae27a305 2024-09-18T15:16:43.933 INFO:tasks.workunit.client.0.smithi046.stdout:add osd3 10ab86bd-bc7e-4208-a5b7-99e6ae27a305 2024-09-18T15:16:43.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd3 10ab86bd-bc7e-4208-a5b7-99e6ae27a305' 2024-09-18T15:16:43.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T15:16:43.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQDb7upmBP2ROBAAkb67GVAoT+8cAN1E2T1D9w== 2024-09-18T15:16:43.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQDb7upmBP2ROBAAkb67GVAoT+8cAN1E2T1D9w=="}' 2024-09-18T15:16:43.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 10ab86bd-bc7e-4208-a5b7-99e6ae27a305 -i td/osd-scrub-repair/2/new.json 2024-09-18T15:16:44.374 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:16:44.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/2/new.json 2024-09-18T15:16:44.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDb7upmBP2ROBAAkb67GVAoT+8cAN1E2T1D9w== --osd-uuid 10ab86bd-bc7e-4208-a5b7-99e6ae27a305 --osd-objectstore=filestore 2024-09-18T15:16:44.426 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:44.418+0000 7fbe97b2d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:44.466 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:44.458+0000 7fbe97b2d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:44.468 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:44.458+0000 7fbe97b2d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:45.002 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:44.994+0000 7fbe97b2d540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:16:45.524 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:45.518+0000 7fbe97b2d540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:16:45.524 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:45.518+0000 7fbe97b2d540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:16:45.524 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:45.518+0000 7fbe97b2d540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:16:46.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-18T15:16:46.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T15:16:46.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd2 key to auth repository 2024-09-18T15:16:46.210 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T15:16:46.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:16:46.589 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: key for osd.2 not found in provided keyring 2024-09-18T15:16:46.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.2 2024-09-18T15:16:46.593 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T15:16:46.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:16:46.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T15:16:46.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T15:16:46.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T15:16:46.616 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:46.610+0000 7fea7c29a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:46.619 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:46.610+0000 7fea7c29a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:46.621 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:46.614+0000 7fea7c29a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:46.623 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:46.614+0000 7fea7c29a540 -1 Falling back to public interface 2024-09-18T15:16:46.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 2 2024-09-18T15:16:46.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:16:46.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:16:46.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:16:46.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:16:46.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:46.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:16:46.996 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:16:46.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:46.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:16:47.183 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:47.174+0000 7fea7c29a540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:16:47.184 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:47.174+0000 7fea7c29a540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:16:47.184 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:47.174+0000 7fea7c29a540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:16:47.191 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:47.182+0000 7fea7c29a540 -1 osd.2 0 log_to_monitors true 2024-09-18T15:16:47.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:48.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:48.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:48.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:16:48.353 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:16:48.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:48.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:16:48.745 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/256071,v1:127.0.0.1:6819/256071] [v2:127.0.0.1:6820/256071,v1:127.0.0.1:6821/256071] exists,up 10ab86bd-bc7e-4208-a5b7-99e6ae27a305 2024-09-18T15:16:48.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:16:48.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:16:48.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:16:48.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-18T15:16:48.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: '[' false = true ']' 2024-09-18T15:16:48.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:840: corrupt_and_repair_lrc: run_osd_filestore td/osd-scrub-repair 3 2024-09-18T15:16:48.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T15:16:48.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T15:16:48.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=3 2024-09-18T15:16:48.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T15:16:48.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:16:48.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:16:48.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:16:48.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:16:48.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:16:48.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-18T15:16:48.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-18T15:16:48.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T15:16:48.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T15:16:48.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:16:48.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T15:16:48.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:16:48.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:16:48.750 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:16:48.750 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:16:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:16:48.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:16:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:16:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T15:16:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T15:16:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T15:16:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:16:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:16:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:16:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:16:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:16:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T15:16:48.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T15:16:48.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/3 2024-09-18T15:16:48.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T15:16:48.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=0fcca98d-f3b0-4b6a-8ec6-ccc43b7d6de2 2024-09-18T15:16:48.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd3 0fcca98d-f3b0-4b6a-8ec6-ccc43b7d6de2' 2024-09-18T15:16:48.754 INFO:tasks.workunit.client.0.smithi046.stdout:add osd3 0fcca98d-f3b0-4b6a-8ec6-ccc43b7d6de2 2024-09-18T15:16:48.754 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T15:16:48.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQDg7upmElDsLRAAemy2Bd4Zj4JpebJZfdTEnQ== 2024-09-18T15:16:48.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQDg7upmElDsLRAAemy2Bd4Zj4JpebJZfdTEnQ=="}' 2024-09-18T15:16:48.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 0fcca98d-f3b0-4b6a-8ec6-ccc43b7d6de2 -i td/osd-scrub-repair/3/new.json 2024-09-18T15:16:49.217 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:16:49.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/3/new.json 2024-09-18T15:16:49.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDg7upmElDsLRAAemy2Bd4Zj4JpebJZfdTEnQ== --osd-uuid 0fcca98d-f3b0-4b6a-8ec6-ccc43b7d6de2 --osd-objectstore=filestore 2024-09-18T15:16:49.269 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:49.262+0000 7f1df1b33540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:49.272 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:49.266+0000 7f1df1b33540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:49.274 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:49.266+0000 7f1df1b33540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:50.029 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:50.022+0000 7f1df1b33540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:16:50.693 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:50.686+0000 7f1df1b33540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:16:50.693 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:50.686+0000 7f1df1b33540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:16:50.693 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:50.686+0000 7f1df1b33540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:16:51.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-18T15:16:51.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T15:16:51.463 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd3 key to auth repository 2024-09-18T15:16:51.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd3 key to auth repository 2024-09-18T15:16:51.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:16:51.857 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.3 2024-09-18T15:16:51.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.3 2024-09-18T15:16:51.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T15:16:51.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:16:51.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T15:16:51.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T15:16:51.889 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:51.882+0000 7f46352c6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:51.931 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:51.922+0000 7f46352c6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:51.933 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:51.926+0000 7f46352c6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:51.935 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:51.926+0000 7f46352c6540 -1 Falling back to public interface 2024-09-18T15:16:52.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 3 2024-09-18T15:16:52.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:16:52.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:16:52.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:16:52.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:16:52.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:52.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:16:52.195 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:16:52.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:52.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:16:52.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:52.830 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:52.821+0000 7f46352c6540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:16:52.834 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:52.825+0000 7f46352c6540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:16:52.835 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:52.825+0000 7f46352c6540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:16:52.873 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:52.833+0000 7f46352c6540 -1 osd.3 0 log_to_monitors true 2024-09-18T15:16:53.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:53.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:53.662 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:16:53.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:16:53.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:53.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:16:54.018 INFO:tasks.workunit.client.0.smithi046.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/256356,v1:127.0.0.1:6827/256356] [v2:127.0.0.1:6828/256356,v1:127.0.0.1:6829/256356] exists,up 0fcca98d-f3b0-4b6a-8ec6-ccc43b7d6de2 2024-09-18T15:16:54.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:16:54.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:16:54.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:16:54.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-18T15:16:54.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: '[' false = true ']' 2024-09-18T15:16:54.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:840: corrupt_and_repair_lrc: run_osd_filestore td/osd-scrub-repair 4 2024-09-18T15:16:54.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T15:16:54.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T15:16:54.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=4 2024-09-18T15:16:54.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T15:16:54.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/4 2024-09-18T15:16:54.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:16:54.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:16:54.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:16:54.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:16:54.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-09-18T15:16:54.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-09-18T15:16:54.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T15:16:54.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T15:16:54.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:16:54.027 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T15:16:54.027 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:16:54.027 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:16:54.027 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:16:54.027 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:16:54.027 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:16:54.028 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:16:54.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:16:54.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T15:16:54.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T15:16:54.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T15:16:54.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:16:54.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:16:54.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:16:54.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:16:54.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:16:54.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T15:16:54.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T15:16:54.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/4 2024-09-18T15:16:54.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T15:16:54.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=544c6fe8-7151-4166-97a7-c04b59e9c2e2 2024-09-18T15:16:54.029 INFO:tasks.workunit.client.0.smithi046.stdout:add osd3 544c6fe8-7151-4166-97a7-c04b59e9c2e2 2024-09-18T15:16:54.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd3 544c6fe8-7151-4166-97a7-c04b59e9c2e2' 2024-09-18T15:16:54.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T15:16:54.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQDm7upmG2RpAhAAta4nHWMjyQNcJDIJZvcDyQ== 2024-09-18T15:16:54.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQDm7upmG2RpAhAAta4nHWMjyQNcJDIJZvcDyQ=="}' 2024-09-18T15:16:54.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 544c6fe8-7151-4166-97a7-c04b59e9c2e2 -i td/osd-scrub-repair/4/new.json 2024-09-18T15:16:54.438 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:16:54.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/4/new.json 2024-09-18T15:16:54.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 4 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDm7upmG2RpAhAAta4nHWMjyQNcJDIJZvcDyQ== --osd-uuid 544c6fe8-7151-4166-97a7-c04b59e9c2e2 --osd-objectstore=filestore 2024-09-18T15:16:54.490 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:54.481+0000 7f6c9af11540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:54.494 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:54.485+0000 7f6c9af11540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:54.496 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:54.485+0000 7f6c9af11540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:55.204 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:55.193+0000 7f6c9af11540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:16:55.920 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:55.913+0000 7f6c9af11540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:16:55.945 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:55.937+0000 7f6c9af11540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:16:55.945 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:55.937+0000 7f6c9af11540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:16:56.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/4/keyring 2024-09-18T15:16:56.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T15:16:56.710 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd4 key to auth repository 2024-09-18T15:16:56.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd4 key to auth repository 2024-09-18T15:16:56.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:16:57.097 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: key for osd.4 not found in provided keyring 2024-09-18T15:16:57.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.4 2024-09-18T15:16:57.100 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.4 2024-09-18T15:16:57.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 4 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:16:57.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T15:16:57.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T15:16:57.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T15:16:57.123 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:57.113+0000 7f29c41bd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:57.128 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:57.117+0000 7f29c41bd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:57.129 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:57.121+0000 7f29c41bd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:57.131 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:57.121+0000 7f29c41bd540 -1 Falling back to public interface 2024-09-18T15:16:57.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 4 2024-09-18T15:16:57.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:16:57.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-18T15:16:57.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:16:57.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:16:57.518 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:16:57.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:57.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:16:57.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:57.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:16:57.857 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:57.849+0000 7f29c41bd540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:16:57.862 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:57.853+0000 7f29c41bd540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:16:57.862 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:57.853+0000 7f29c41bd540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:16:57.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:57.872 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:16:57.861+0000 7f29c41bd540 -1 osd.4 0 log_to_monitors true 2024-09-18T15:16:58.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:58.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:58.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:16:58.868 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:16:58.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:58.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:16:59.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:00.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:00.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:00.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:17:00.261 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:17:00.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:00.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:17:00.570 INFO:tasks.workunit.client.0.smithi046.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/256638,v1:127.0.0.1:6835/256638] [v2:127.0.0.1:6836/256638,v1:127.0.0.1:6837/256638] exists,up 544c6fe8-7151-4166-97a7-c04b59e9c2e2 2024-09-18T15:17:00.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:17:00.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:17:00.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:17:00.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-18T15:17:00.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: '[' false = true ']' 2024-09-18T15:17:00.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:840: corrupt_and_repair_lrc: run_osd_filestore td/osd-scrub-repair 5 2024-09-18T15:17:00.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T15:17:00.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T15:17:00.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=5 2024-09-18T15:17:00.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T15:17:00.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/5 2024-09-18T15:17:00.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:17:00.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:17:00.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:17:00.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:17:00.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-18T15:17:00.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-18T15:17:00.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T15:17:00.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T15:17:00.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:17:00.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T15:17:00.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:17:00.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:17:00.575 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:17:00.575 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:17:00.575 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:17:00.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:17:00.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:17:00.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T15:17:00.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T15:17:00.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T15:17:00.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:17:00.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:17:00.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:17:00.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:17:00.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:17:00.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T15:17:00.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T15:17:00.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/5 2024-09-18T15:17:00.578 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T15:17:00.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=581d8192-9e17-497f-95a0-795d5338b5fe 2024-09-18T15:17:00.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd3 581d8192-9e17-497f-95a0-795d5338b5fe' 2024-09-18T15:17:00.579 INFO:tasks.workunit.client.0.smithi046.stdout:add osd3 581d8192-9e17-497f-95a0-795d5338b5fe 2024-09-18T15:17:00.580 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T15:17:00.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQDs7upmpfJPIxAAXN0E/0xz+S+OkIO0372vVg== 2024-09-18T15:17:00.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQDs7upmpfJPIxAAXN0E/0xz+S+OkIO0372vVg=="}' 2024-09-18T15:17:00.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 581d8192-9e17-497f-95a0-795d5338b5fe -i td/osd-scrub-repair/5/new.json 2024-09-18T15:17:00.988 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:17:01.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/5/new.json 2024-09-18T15:17:01.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 5 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDs7upmpfJPIxAAXN0E/0xz+S+OkIO0372vVg== --osd-uuid 581d8192-9e17-497f-95a0-795d5338b5fe --osd-objectstore=filestore 2024-09-18T15:17:01.041 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:01.033+0000 7f7d63d23540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:01.045 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:01.037+0000 7f7d63d23540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:01.047 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:01.037+0000 7f7d63d23540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:01.647 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:01.641+0000 7f7d63d23540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:17:02.360 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:02.349+0000 7f7d63d23540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:17:02.360 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:02.353+0000 7f7d63d23540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:17:02.360 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:02.353+0000 7f7d63d23540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:17:03.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/5/keyring 2024-09-18T15:17:03.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T15:17:03.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd5 key to auth repository 2024-09-18T15:17:03.318 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd5 key to auth repository 2024-09-18T15:17:03.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:17:03.698 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: key for osd.5 not found in provided keyring 2024-09-18T15:17:03.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.5 2024-09-18T15:17:03.702 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.5 2024-09-18T15:17:03.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T15:17:03.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 5 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:17:03.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T15:17:03.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T15:17:03.731 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:03.721+0000 7fb16ec16540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:03.734 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:03.725+0000 7fb16ec16540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:03.740 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:03.725+0000 7fb16ec16540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:03.741 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:03.729+0000 7fb16ec16540 -1 Falling back to public interface 2024-09-18T15:17:04.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 5 2024-09-18T15:17:04.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:17:04.007 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:17:04.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T15:17:04.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:17:04.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:17:04.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:04.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:17:04.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:04.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:17:04.256 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:04.245+0000 7fb16ec16540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:17:04.262 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:04.253+0000 7fb16ec16540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:17:04.262 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:04.253+0000 7fb16ec16540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:17:04.273 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:04.265+0000 7fb16ec16540 -1 osd.5 0 log_to_monitors true 2024-09-18T15:17:04.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:05.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:05.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:05.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:17:05.357 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:17:05.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:05.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:17:05.698 INFO:tasks.workunit.client.0.smithi046.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/256949,v1:127.0.0.1:6843/256949] [v2:127.0.0.1:6844/256949,v1:127.0.0.1:6845/256949] exists,up 581d8192-9e17-497f-95a0-795d5338b5fe 2024-09-18T15:17:05.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:17:05.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:17:05.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:17:05.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-18T15:17:05.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: '[' false = true ']' 2024-09-18T15:17:05.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:840: corrupt_and_repair_lrc: run_osd_filestore td/osd-scrub-repair 6 2024-09-18T15:17:05.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T15:17:05.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T15:17:05.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=6 2024-09-18T15:17:05.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T15:17:05.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/6 2024-09-18T15:17:05.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:17:05.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:17:05.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:17:05.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:17:05.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-09-18T15:17:05.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-09-18T15:17:05.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T15:17:05.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T15:17:05.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:17:05.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T15:17:05.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:17:05.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:17:05.703 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:17:05.703 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:17:05.703 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:17:05.703 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:17:05.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:17:05.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T15:17:05.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T15:17:05.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T15:17:05.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:17:05.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:17:05.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:17:05.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:17:05.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:17:05.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T15:17:05.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T15:17:05.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/6 2024-09-18T15:17:05.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T15:17:05.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=ddcb2109-68a8-4df2-b195-e81b82e73b44 2024-09-18T15:17:05.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd3 ddcb2109-68a8-4df2-b195-e81b82e73b44' 2024-09-18T15:17:05.706 INFO:tasks.workunit.client.0.smithi046.stdout:add osd3 ddcb2109-68a8-4df2-b195-e81b82e73b44 2024-09-18T15:17:05.707 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T15:17:05.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQDx7upmPGcwKxAAJ6EzB8Xd5tSgRscT28Fn0A== 2024-09-18T15:17:05.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQDx7upmPGcwKxAAJ6EzB8Xd5tSgRscT28Fn0A=="}' 2024-09-18T15:17:05.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new ddcb2109-68a8-4df2-b195-e81b82e73b44 -i td/osd-scrub-repair/6/new.json 2024-09-18T15:17:06.117 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:17:06.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/6/new.json 2024-09-18T15:17:06.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 6 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDx7upmPGcwKxAAJ6EzB8Xd5tSgRscT28Fn0A== --osd-uuid ddcb2109-68a8-4df2-b195-e81b82e73b44 --osd-objectstore=filestore 2024-09-18T15:17:06.172 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:06.161+0000 7f68c9777540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:06.174 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:06.165+0000 7f68c9777540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:06.176 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:06.169+0000 7f68c9777540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:06.802 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:06.793+0000 7f68c9777540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:17:07.659 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:07.649+0000 7f68c9777540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:17:07.659 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:07.649+0000 7f68c9777540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:17:07.659 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:07.649+0000 7f68c9777540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:17:08.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/6/keyring 2024-09-18T15:17:08.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T15:17:08.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd6 key to auth repository 2024-09-18T15:17:08.486 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd6 key to auth repository 2024-09-18T15:17:08.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:17:08.866 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: key for osd.6 not found in provided keyring 2024-09-18T15:17:08.869 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.6 2024-09-18T15:17:08.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.6 2024-09-18T15:17:08.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 6 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:17:08.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T15:17:08.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T15:17:08.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T15:17:08.894 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:08.885+0000 7f1ad61b6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:08.896 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:08.889+0000 7f1ad61b6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:08.900 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:08.889+0000 7f1ad61b6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:08.900 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:08.889+0000 7f1ad61b6540 -1 Falling back to public interface 2024-09-18T15:17:09.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 6 2024-09-18T15:17:09.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:17:09.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-09-18T15:17:09.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:17:09.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:17:09.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:09.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:17:09.255 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:17:09.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T15:17:09.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:09.594 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:09.585+0000 7f1ad61b6540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:17:09.595 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:09.585+0000 7f1ad61b6540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:17:09.595 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:09.585+0000 7f1ad61b6540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:17:09.605 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:09.597+0000 7f1ad61b6540 -1 osd.6 0 log_to_monitors true 2024-09-18T15:17:09.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:10.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:10.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:10.614 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:17:10.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:17:10.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:10.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T15:17:10.932 INFO:tasks.workunit.client.0.smithi046.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/257232,v1:127.0.0.1:6851/257232] [v2:127.0.0.1:6852/257232,v1:127.0.0.1:6853/257232] exists,up ddcb2109-68a8-4df2-b195-e81b82e73b44 2024-09-18T15:17:10.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:17:10.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:17:10.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:17:10.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-18T15:17:10.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: '[' false = true ']' 2024-09-18T15:17:10.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:840: corrupt_and_repair_lrc: run_osd_filestore td/osd-scrub-repair 7 2024-09-18T15:17:10.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T15:17:10.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T15:17:10.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=7 2024-09-18T15:17:10.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T15:17:10.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/7 2024-09-18T15:17:10.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:17:10.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:17:10.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:17:10.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:17:10.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-09-18T15:17:10.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-09-18T15:17:10.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T15:17:10.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T15:17:10.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:17:10.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T15:17:10.936 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:17:10.936 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:17:10.937 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:17:10.937 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:17:10.937 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:17:10.937 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:17:10.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:17:10.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T15:17:10.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T15:17:10.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T15:17:10.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:17:10.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:17:10.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:17:10.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:17:10.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:17:10.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T15:17:10.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T15:17:10.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/7 2024-09-18T15:17:10.939 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T15:17:10.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=05945907-db09-4e5c-83cb-832cb307a0df 2024-09-18T15:17:10.940 INFO:tasks.workunit.client.0.smithi046.stdout:add osd3 05945907-db09-4e5c-83cb-832cb307a0df 2024-09-18T15:17:10.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd3 05945907-db09-4e5c-83cb-832cb307a0df' 2024-09-18T15:17:10.940 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T15:17:10.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQD27upmOibLOBAALQ48pFKSpMZ2I1heVEhiOw== 2024-09-18T15:17:10.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQD27upmOibLOBAALQ48pFKSpMZ2I1heVEhiOw=="}' 2024-09-18T15:17:10.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 05945907-db09-4e5c-83cb-832cb307a0df -i td/osd-scrub-repair/7/new.json 2024-09-18T15:17:11.423 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:17:11.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/7/new.json 2024-09-18T15:17:11.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 7 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD27upmOibLOBAALQ48pFKSpMZ2I1heVEhiOw== --osd-uuid 05945907-db09-4e5c-83cb-832cb307a0df --osd-objectstore=filestore 2024-09-18T15:17:11.478 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:11.469+0000 7fd0670fb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:11.481 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:11.473+0000 7fd0670fb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:11.482 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:11.473+0000 7fd0670fb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:12.050 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:12.041+0000 7fd0670fb540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:17:12.731 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:12.721+0000 7fd0670fb540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:17:12.738 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:12.729+0000 7fd0670fb540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:17:12.738 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:12.729+0000 7fd0670fb540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:17:13.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/7/keyring 2024-09-18T15:17:13.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T15:17:13.718 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd7 key to auth repository 2024-09-18T15:17:13.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd7 key to auth repository 2024-09-18T15:17:13.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:17:14.091 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: key for osd.7 not found in provided keyring 2024-09-18T15:17:14.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.7 2024-09-18T15:17:14.094 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.7 2024-09-18T15:17:14.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 7 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:17:14.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T15:17:14.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T15:17:14.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T15:17:14.118 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:14.109+0000 7f61c4016540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:14.122 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:14.113+0000 7f61c4016540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:14.128 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:14.113+0000 7f61c4016540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:14.128 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:14.117+0000 7f61c4016540 -1 Falling back to public interface 2024-09-18T15:17:14.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 7 2024-09-18T15:17:14.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:17:14.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-09-18T15:17:14.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:17:14.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:17:14.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:14.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:17:14.464 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:17:14.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:14.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T15:17:14.934 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:14.925+0000 7f61c4016540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:17:14.935 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:14.925+0000 7f61c4016540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:17:14.935 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:14.925+0000 7f61c4016540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:17:14.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:14.945 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:14.937+0000 7f61c4016540 -1 osd.7 0 log_to_monitors true 2024-09-18T15:17:15.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:15.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:15.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:17:15.942 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:17:15.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:15.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T15:17:16.327 INFO:tasks.workunit.client.0.smithi046.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/257515,v1:127.0.0.1:6859/257515] [v2:127.0.0.1:6860/257515,v1:127.0.0.1:6861/257515] exists,up 05945907-db09-4e5c-83cb-832cb307a0df 2024-09-18T15:17:16.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:17:16.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:17:16.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:17:16.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-18T15:17:16.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: '[' false = true ']' 2024-09-18T15:17:16.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:840: corrupt_and_repair_lrc: run_osd_filestore td/osd-scrub-repair 8 2024-09-18T15:17:16.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T15:17:16.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T15:17:16.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=8 2024-09-18T15:17:16.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T15:17:16.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/8 2024-09-18T15:17:16.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:17:16.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:17:16.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:17:16.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:17:16.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-09-18T15:17:16.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-09-18T15:17:16.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T15:17:16.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T15:17:16.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:17:16.331 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T15:17:16.331 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:17:16.331 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:17:16.333 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:17:16.333 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:17:16.333 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:17:16.333 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:17:16.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:17:16.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T15:17:16.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T15:17:16.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T15:17:16.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:17:16.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:17:16.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:17:16.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:17:16.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:17:16.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T15:17:16.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T15:17:16.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/8 2024-09-18T15:17:16.334 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T15:17:16.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=9f8bd111-bf51-40ac-9c43-a81ac9fdb9b0 2024-09-18T15:17:16.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd3 9f8bd111-bf51-40ac-9c43-a81ac9fdb9b0' 2024-09-18T15:17:16.336 INFO:tasks.workunit.client.0.smithi046.stdout:add osd3 9f8bd111-bf51-40ac-9c43-a81ac9fdb9b0 2024-09-18T15:17:16.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T15:17:16.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQD87upmP58TFRAAEPZPg3Jvkv4qogGDsr4mXA== 2024-09-18T15:17:16.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQD87upmP58TFRAAEPZPg3Jvkv4qogGDsr4mXA=="}' 2024-09-18T15:17:16.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 9f8bd111-bf51-40ac-9c43-a81ac9fdb9b0 -i td/osd-scrub-repair/8/new.json 2024-09-18T15:17:16.748 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:17:16.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/8/new.json 2024-09-18T15:17:16.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 8 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD87upmP58TFRAAEPZPg3Jvkv4qogGDsr4mXA== --osd-uuid 9f8bd111-bf51-40ac-9c43-a81ac9fdb9b0 --osd-objectstore=filestore 2024-09-18T15:17:16.804 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:16.793+0000 7f28d33ff540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:16.866 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:16.857+0000 7f28d33ff540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:16.869 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:16.861+0000 7f28d33ff540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:17.424 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:17.417+0000 7f28d33ff540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:17:18.071 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:18.061+0000 7f28d33ff540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:17:18.080 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:18.069+0000 7f28d33ff540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:17:18.080 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:18.069+0000 7f28d33ff540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:17:19.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/8/keyring 2024-09-18T15:17:19.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T15:17:19.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd8 key to auth repository 2024-09-18T15:17:19.144 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd8 key to auth repository 2024-09-18T15:17:19.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:17:19.542 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: key for osd.8 not found in provided keyring 2024-09-18T15:17:19.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.8 2024-09-18T15:17:19.545 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.8 2024-09-18T15:17:19.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T15:17:19.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 8 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:17:19.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T15:17:19.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T15:17:19.575 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:19.565+0000 7f0295f18540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:19.577 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:19.569+0000 7f0295f18540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:19.578 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:19.569+0000 7f0295f18540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:19.588 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:19.577+0000 7f0295f18540 -1 Falling back to public interface 2024-09-18T15:17:19.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 8 2024-09-18T15:17:19.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:17:19.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-09-18T15:17:19.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:17:19.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:17:19.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:19.839 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:17:19.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:17:19.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:19.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T15:17:20.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:20.383 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:20.373+0000 7f0295f18540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:17:20.384 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:20.373+0000 7f0295f18540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:17:20.384 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:20.373+0000 7f0295f18540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:17:20.393 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:20.385+0000 7f0295f18540 -1 osd.8 0 log_to_monitors true 2024-09-18T15:17:21.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:21.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:21.365 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:17:21.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:17:21.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:21.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T15:17:21.783 INFO:tasks.workunit.client.0.smithi046.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/257798,v1:127.0.0.1:6867/257798] [v2:127.0.0.1:6868/257798,v1:127.0.0.1:6869/257798] exists,up 9f8bd111-bf51-40ac-9c43-a81ac9fdb9b0 2024-09-18T15:17:21.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:17:21.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:17:21.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:17:21.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-18T15:17:21.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: '[' false = true ']' 2024-09-18T15:17:21.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:840: corrupt_and_repair_lrc: run_osd_filestore td/osd-scrub-repair 9 2024-09-18T15:17:21.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T15:17:21.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T15:17:21.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=9 2024-09-18T15:17:21.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T15:17:21.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/9 2024-09-18T15:17:21.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:17:21.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:17:21.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:17:21.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:17:21.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-18T15:17:21.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-18T15:17:21.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T15:17:21.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T15:17:21.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:17:21.787 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T15:17:21.787 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:17:21.788 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:17:21.788 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:17:21.788 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:17:21.788 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:17:21.788 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:17:21.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:17:21.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T15:17:21.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T15:17:21.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T15:17:21.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:17:21.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:17:21.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:17:21.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:17:21.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:17:21.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T15:17:21.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T15:17:21.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/9 2024-09-18T15:17:21.791 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T15:17:21.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=57987690-7a54-4475-bda5-ffdec9be4750 2024-09-18T15:17:21.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd3 57987690-7a54-4475-bda5-ffdec9be4750' 2024-09-18T15:17:21.792 INFO:tasks.workunit.client.0.smithi046.stdout:add osd3 57987690-7a54-4475-bda5-ffdec9be4750 2024-09-18T15:17:21.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T15:17:21.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQAB7+pmNlwhMBAAWKMOnIFvq6ZbDRf1yRVmow== 2024-09-18T15:17:21.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQAB7+pmNlwhMBAAWKMOnIFvq6ZbDRf1yRVmow=="}' 2024-09-18T15:17:21.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 57987690-7a54-4475-bda5-ffdec9be4750 -i td/osd-scrub-repair/9/new.json 2024-09-18T15:17:22.207 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:17:22.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/9/new.json 2024-09-18T15:17:22.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 9 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAB7+pmNlwhMBAAWKMOnIFvq6ZbDRf1yRVmow== --osd-uuid 57987690-7a54-4475-bda5-ffdec9be4750 --osd-objectstore=filestore 2024-09-18T15:17:22.257 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:22.249+0000 7f516b8e7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:22.260 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:22.253+0000 7f516b8e7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:22.261 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:22.253+0000 7f516b8e7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:22.902 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:22.893+0000 7f516b8e7540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:17:23.759 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:23.749+0000 7f516b8e7540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:17:23.759 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:23.749+0000 7f516b8e7540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:17:23.759 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:23.749+0000 7f516b8e7540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:17:24.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/9/keyring 2024-09-18T15:17:24.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T15:17:24.767 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd9 key to auth repository 2024-09-18T15:17:24.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd9 key to auth repository 2024-09-18T15:17:24.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:17:25.140 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: key for osd.9 not found in provided keyring 2024-09-18T15:17:25.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.9 2024-09-18T15:17:25.143 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.9 2024-09-18T15:17:25.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 9 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:17:25.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T15:17:25.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T15:17:25.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T15:17:25.168 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:25.157+0000 7f25c7705540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:25.170 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:25.161+0000 7f25c7705540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:25.171 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:25.161+0000 7f25c7705540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:25.181 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:25.173+0000 7f25c7705540 -1 Falling back to public interface 2024-09-18T15:17:25.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 9 2024-09-18T15:17:25.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:17:25.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-09-18T15:17:25.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:17:25.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:17:25.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:25.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:17:25.498 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:17:25.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:25.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:17:26.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:26.145 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:26.137+0000 7f25c7705540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:17:26.151 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:26.141+0000 7f25c7705540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:17:26.152 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:26.141+0000 7f25c7705540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:17:26.179 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:26.153+0000 7f25c7705540 -1 osd.9 0 log_to_monitors true 2024-09-18T15:17:27.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:27.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:27.117 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:17:27.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:17:27.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:27.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:17:27.494 INFO:tasks.workunit.client.0.smithi046.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/258081,v1:127.0.0.1:6875/258081] [v2:127.0.0.1:6876/258081,v1:127.0.0.1:6877/258081] exists,up 57987690-7a54-4475-bda5-ffdec9be4750 2024-09-18T15:17:27.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:17:27.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:17:27.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:17:27.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:843: corrupt_and_repair_lrc: create_rbd_pool 2024-09-18T15:17:27.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T15:17:27.831 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' does not exist 2024-09-18T15:17:27.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T15:17:27.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T15:17:28.319 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' created 2024-09-18T15:17:28.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:17:29.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T15:17:33.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:844: corrupt_and_repair_lrc: wait_for_clean 2024-09-18T15:17:33.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:17:33.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:17:33.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:17:33.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:17:33.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:17:33.656 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:17:33.656 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:17:33.657 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:17:33.657 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:17:33.657 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:17:33.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:17:33.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:17:33.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:17:33.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:17:33.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:17:34.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:17:34.201 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:17:34.201 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:17:34.201 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:17:34.201 INFO:tasks.workunit.client.0.smithi046.stderr:4 2024-09-18T15:17:34.201 INFO:tasks.workunit.client.0.smithi046.stderr:5 2024-09-18T15:17:34.201 INFO:tasks.workunit.client.0.smithi046.stderr:6 2024-09-18T15:17:34.201 INFO:tasks.workunit.client.0.smithi046.stderr:7 2024-09-18T15:17:34.201 INFO:tasks.workunit.client.0.smithi046.stderr:8 2024-09-18T15:17:34.201 INFO:tasks.workunit.client.0.smithi046.stderr:9' 2024-09-18T15:17:34.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:17:34.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:17:34.202 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:17:34.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836492 2024-09-18T15:17:34.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836492 2024-09-18T15:17:34.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836492' 2024-09-18T15:17:34.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:17:34.351 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:17:34.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672972 2024-09-18T15:17:34.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672972 2024-09-18T15:17:34.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836492 1-42949672972' 2024-09-18T15:17:34.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:17:34.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:17:34.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509452 2024-09-18T15:17:34.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509452 2024-09-18T15:17:34.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836492 1-42949672972 2-64424509452' 2024-09-18T15:17:34.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:17:34.629 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:17:34.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345930 2024-09-18T15:17:34.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345930 2024-09-18T15:17:34.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836492 1-42949672972 2-64424509452 3-85899345930' 2024-09-18T15:17:34.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:17:34.769 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:17:34.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182409 2024-09-18T15:17:34.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182409 2024-09-18T15:17:34.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836492 1-42949672972 2-64424509452 3-85899345930 4-107374182409' 2024-09-18T15:17:34.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:17:34.907 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:17:35.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018887 2024-09-18T15:17:35.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018887 2024-09-18T15:17:35.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836492 1-42949672972 2-64424509452 3-85899345930 4-107374182409 5-128849018887' 2024-09-18T15:17:35.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:17:35.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:17:35.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855366 2024-09-18T15:17:35.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855366 2024-09-18T15:17:35.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836492 1-42949672972 2-64424509452 3-85899345930 4-107374182409 5-128849018887 6-150323855366' 2024-09-18T15:17:35.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:17:35.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:17:35.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691846 2024-09-18T15:17:35.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691846 2024-09-18T15:17:35.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836492 1-42949672972 2-64424509452 3-85899345930 4-107374182409 5-128849018887 6-150323855366 7-171798691846' 2024-09-18T15:17:35.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:17:35.356 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:17:35.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528324 2024-09-18T15:17:35.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528324 2024-09-18T15:17:35.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836492 1-42949672972 2-64424509452 3-85899345930 4-107374182409 5-128849018887 6-150323855366 7-171798691846 8-193273528324' 2024-09-18T15:17:35.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:17:35.490 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:17:35.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=214748364803 2024-09-18T15:17:35.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 214748364803 2024-09-18T15:17:35.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836492 1-42949672972 2-64424509452 3-85899345930 4-107374182409 5-128849018887 6-150323855366 7-171798691846 8-193273528324 9-214748364803' 2024-09-18T15:17:35.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:17:35.632 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836492 2024-09-18T15:17:35.632 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:17:35.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:17:35.635 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:17:35.635 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836492 2024-09-18T15:17:35.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836492 2024-09-18T15:17:35.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-09-18T15:17:35.636 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836492 2024-09-18T15:17:35.637 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:17:35.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836491 -lt 21474836492 2024-09-18T15:17:35.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:17:36.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:17:36.940 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:17:37.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836493 -lt 21474836492 2024-09-18T15:17:37.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:17:37.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672972 2024-09-18T15:17:37.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:17:37.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:17:37.288 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672972 2024-09-18T15:17:37.288 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:17:37.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672972 2024-09-18T15:17:37.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-09-18T15:17:37.290 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672972 2024-09-18T15:17:37.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:17:37.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672972 -lt 42949672972 2024-09-18T15:17:37.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:17:37.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509452 2024-09-18T15:17:37.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:17:37.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:17:37.644 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509452 2024-09-18T15:17:37.644 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:17:37.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509452 2024-09-18T15:17:37.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-18T15:17:37.647 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509452 2024-09-18T15:17:37.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:17:38.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509452 -lt 64424509452 2024-09-18T15:17:38.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:17:38.069 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345930 2024-09-18T15:17:38.069 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:17:38.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:17:38.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345930 2024-09-18T15:17:38.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:17:38.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345930 2024-09-18T15:17:38.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345930' 2024-09-18T15:17:38.075 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.3 seq 85899345930 2024-09-18T15:17:38.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:17:38.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345930 -lt 85899345930 2024-09-18T15:17:38.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:17:38.415 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182409 2024-09-18T15:17:38.415 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:17:38.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:17:38.419 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182409 2024-09-18T15:17:38.419 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:17:38.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182409 2024-09-18T15:17:38.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182409' 2024-09-18T15:17:38.421 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.4 seq 107374182409 2024-09-18T15:17:38.422 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:17:38.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182410 -lt 107374182409 2024-09-18T15:17:38.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:17:38.728 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018887 2024-09-18T15:17:38.728 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:17:38.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:17:38.732 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018887 2024-09-18T15:17:38.732 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:17:38.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018887 2024-09-18T15:17:38.734 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.5 seq 128849018887 2024-09-18T15:17:38.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018887' 2024-09-18T15:17:38.734 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:17:39.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018888 -lt 128849018887 2024-09-18T15:17:39.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:17:39.084 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855366 2024-09-18T15:17:39.084 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:17:39.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:17:39.088 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:17:39.088 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855366 2024-09-18T15:17:39.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855366 2024-09-18T15:17:39.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855366' 2024-09-18T15:17:39.089 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.6 seq 150323855366 2024-09-18T15:17:39.090 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:17:39.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855367 -lt 150323855366 2024-09-18T15:17:39.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:17:39.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691846 2024-09-18T15:17:39.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:17:39.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:17:39.444 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691846 2024-09-18T15:17:39.444 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:17:39.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691846 2024-09-18T15:17:39.445 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.7 seq 171798691846 2024-09-18T15:17:39.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691846' 2024-09-18T15:17:39.446 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:17:39.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691847 -lt 171798691846 2024-09-18T15:17:39.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:17:39.760 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528324 2024-09-18T15:17:39.760 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:17:39.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:17:39.763 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528324 2024-09-18T15:17:39.763 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:17:39.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528324 2024-09-18T15:17:39.765 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.8 seq 193273528324 2024-09-18T15:17:39.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528324' 2024-09-18T15:17:39.766 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:17:40.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528325 -lt 193273528324 2024-09-18T15:17:40.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:17:40.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-214748364803 2024-09-18T15:17:40.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:17:40.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:17:40.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-214748364803 2024-09-18T15:17:40.124 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:17:40.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=214748364803 2024-09-18T15:17:40.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 214748364803' 2024-09-18T15:17:40.126 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.9 seq 214748364803 2024-09-18T15:17:40.127 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:17:40.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 214748364804 -lt 214748364803 2024-09-18T15:17:40.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:17:40.441 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:17:40.441 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:17:40.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T15:17:40.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:17:40.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:17:40.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:17:40.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:17:40.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:17:40.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:17:40.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:17:41.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T15:17:41.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:17:41.183 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:17:41.183 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:17:41.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T15:17:41.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:17:41.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:17:41.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: corrupt_and_repair_lrc: create_ec_pool ecpool false k=4 m=2 l=3 plugin=lrc 2024-09-18T15:17:41.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: create_ec_pool: local pool_name=ecpool 2024-09-18T15:17:41.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: create_ec_pool: shift 2024-09-18T15:17:41.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: create_ec_pool: local allow_overwrites=false 2024-09-18T15:17:41.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: create_ec_pool: shift 2024-09-18T15:17:41.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-09-18T15:17:42.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-18T15:17:42.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-18T15:17:43.622 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'ecpool' created 2024-09-18T15:17:43.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:17:44.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: create_ec_pool: '[' false = true ']' 2024-09-18T15:17:44.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: create_ec_pool: wait_for_clean 2024-09-18T15:17:44.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:17:44.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:17:44.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:17:44.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:17:44.639 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:17:44.640 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:17:44.640 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:17:44.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:17:44.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:17:44.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:17:44.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:17:44.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:17:44.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:17:44.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:17:44.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:17:45.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:17:45.266 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:17:45.267 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:17:45.267 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:17:45.267 INFO:tasks.workunit.client.0.smithi046.stderr:4 2024-09-18T15:17:45.267 INFO:tasks.workunit.client.0.smithi046.stderr:5 2024-09-18T15:17:45.267 INFO:tasks.workunit.client.0.smithi046.stderr:6 2024-09-18T15:17:45.267 INFO:tasks.workunit.client.0.smithi046.stderr:7 2024-09-18T15:17:45.267 INFO:tasks.workunit.client.0.smithi046.stderr:8 2024-09-18T15:17:45.267 INFO:tasks.workunit.client.0.smithi046.stderr:9' 2024-09-18T15:17:45.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:17:45.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:17:45.267 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:17:45.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836495 2024-09-18T15:17:45.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836495 2024-09-18T15:17:45.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836495' 2024-09-18T15:17:45.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:17:45.418 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:17:45.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672975 2024-09-18T15:17:45.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672975 2024-09-18T15:17:45.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836495 1-42949672975' 2024-09-18T15:17:45.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:17:45.556 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:17:45.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509455 2024-09-18T15:17:45.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509455 2024-09-18T15:17:45.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836495 1-42949672975 2-64424509455' 2024-09-18T15:17:45.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:17:45.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:17:45.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345933 2024-09-18T15:17:45.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345933 2024-09-18T15:17:45.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836495 1-42949672975 2-64424509455 3-85899345933' 2024-09-18T15:17:45.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:17:45.826 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:17:45.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182413 2024-09-18T15:17:45.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182413 2024-09-18T15:17:45.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836495 1-42949672975 2-64424509455 3-85899345933 4-107374182413' 2024-09-18T15:17:45.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:17:45.969 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:17:46.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018891 2024-09-18T15:17:46.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018891 2024-09-18T15:17:46.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836495 1-42949672975 2-64424509455 3-85899345933 4-107374182413 5-128849018891' 2024-09-18T15:17:46.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:17:46.106 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:17:46.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855370 2024-09-18T15:17:46.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855370 2024-09-18T15:17:46.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836495 1-42949672975 2-64424509455 3-85899345933 4-107374182413 5-128849018891 6-150323855370' 2024-09-18T15:17:46.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:17:46.243 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:17:46.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691850 2024-09-18T15:17:46.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691850 2024-09-18T15:17:46.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836495 1-42949672975 2-64424509455 3-85899345933 4-107374182413 5-128849018891 6-150323855370 7-171798691850' 2024-09-18T15:17:46.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:17:46.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:17:46.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528327 2024-09-18T15:17:46.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528327 2024-09-18T15:17:46.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836495 1-42949672975 2-64424509455 3-85899345933 4-107374182413 5-128849018891 6-150323855370 7-171798691850 8-193273528327' 2024-09-18T15:17:46.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:17:46.526 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:17:46.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=214748364806 2024-09-18T15:17:46.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 214748364806 2024-09-18T15:17:46.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836495 1-42949672975 2-64424509455 3-85899345933 4-107374182413 5-128849018891 6-150323855370 7-171798691850 8-193273528327 9-214748364806' 2024-09-18T15:17:46.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:17:46.663 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836495 2024-09-18T15:17:46.664 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:17:46.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:17:46.667 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836495 2024-09-18T15:17:46.667 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:17:46.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836495 2024-09-18T15:17:46.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-09-18T15:17:46.669 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836495 2024-09-18T15:17:46.669 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:17:47.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836496 -lt 21474836495 2024-09-18T15:17:47.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:17:47.051 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672975 2024-09-18T15:17:47.052 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:17:47.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:17:47.055 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672975 2024-09-18T15:17:47.055 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:17:47.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672975 2024-09-18T15:17:47.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-18T15:17:47.058 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672975 2024-09-18T15:17:47.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:17:47.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672975 -lt 42949672975 2024-09-18T15:17:47.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:17:47.504 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509455 2024-09-18T15:17:47.504 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:17:47.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:17:47.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509455 2024-09-18T15:17:47.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:17:47.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509455 2024-09-18T15:17:47.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-18T15:17:47.510 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509455 2024-09-18T15:17:47.511 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:17:47.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-18T15:17:47.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:17:47.861 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345933 2024-09-18T15:17:47.861 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:17:47.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:17:47.864 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345933 2024-09-18T15:17:47.864 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:17:47.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345933 2024-09-18T15:17:47.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345933' 2024-09-18T15:17:47.867 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.3 seq 85899345933 2024-09-18T15:17:47.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:17:48.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345933 -lt 85899345933 2024-09-18T15:17:48.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:17:48.225 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182413 2024-09-18T15:17:48.226 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:17:48.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:17:48.229 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182413 2024-09-18T15:17:48.229 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:17:48.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182413 2024-09-18T15:17:48.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182413' 2024-09-18T15:17:48.232 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.4 seq 107374182413 2024-09-18T15:17:48.232 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:17:48.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182413 -lt 107374182413 2024-09-18T15:17:48.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:17:48.661 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018891 2024-09-18T15:17:48.661 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:17:48.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:17:48.664 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018891 2024-09-18T15:17:48.665 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:17:48.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018891 2024-09-18T15:17:48.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018891' 2024-09-18T15:17:48.668 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.5 seq 128849018891 2024-09-18T15:17:48.668 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:17:49.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018891 -lt 128849018891 2024-09-18T15:17:49.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:17:49.016 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855370 2024-09-18T15:17:49.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:17:49.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:17:49.020 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855370 2024-09-18T15:17:49.021 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:17:49.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855370 2024-09-18T15:17:49.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855370' 2024-09-18T15:17:49.024 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.6 seq 150323855370 2024-09-18T15:17:49.024 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:17:49.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855370 -lt 150323855370 2024-09-18T15:17:49.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:17:49.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691850 2024-09-18T15:17:49.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:17:49.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:17:49.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691850 2024-09-18T15:17:49.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:17:49.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691850 2024-09-18T15:17:49.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691850' 2024-09-18T15:17:49.364 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.7 seq 171798691850 2024-09-18T15:17:49.364 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:17:49.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691850 -lt 171798691850 2024-09-18T15:17:49.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:17:49.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528327 2024-09-18T15:17:49.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:17:49.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:17:49.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528327 2024-09-18T15:17:49.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:17:49.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528327 2024-09-18T15:17:49.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528327' 2024-09-18T15:17:49.704 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.8 seq 193273528327 2024-09-18T15:17:49.704 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:17:50.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528328 -lt 193273528327 2024-09-18T15:17:50.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:17:50.046 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-214748364806 2024-09-18T15:17:50.046 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:17:50.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:17:50.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-214748364806 2024-09-18T15:17:50.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:17:50.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=214748364806 2024-09-18T15:17:50.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 214748364806' 2024-09-18T15:17:50.052 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.9 seq 214748364806 2024-09-18T15:17:50.052 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:17:50.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 214748364807 -lt 214748364806 2024-09-18T15:17:50.402 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:17:50.403 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:17:50.403 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:17:50.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:17:50.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:17:50.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:17:50.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:17:50.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:17:50.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:17:50.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:17:50.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:17:51.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:17:51.117 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:17:51.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:17:51.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:17:51.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:17:51.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:17:51.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:17:51.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: create_ec_pool: return 0 2024-09-18T15:17:51.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-09-18T15:17:51.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-18T15:17:51.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-09-18T15:17:51.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:250: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-18T15:17:51.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:17:51.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2024-09-18T15:17:51.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=SOMETHING 2024-09-18T15:17:51.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:17:51.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:17:51.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:17:51.965 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:17:51.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:17:52.848 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:17:52.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:17:52.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:17:52.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-18T15:17:53.172 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:252: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-09-18T15:17:53.172 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-09-18T15:17:53.172 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T15:17:53.173 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:17:53.173 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:17:53.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:252: corrupt_and_repair_erasure_coded: local primary=3 2024-09-18T15:17:53.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: osds=($(get_osds $poolname SOMETHING | sed -e "s/$primary//")) 2024-09-18T15:17:53.974 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-09-18T15:17:53.974 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-09-18T15:17:53.974 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-09-18T15:17:53.974 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-18T15:17:53.976 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:17:53.976 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:17:54.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:17:54.304 INFO:tasks.workunit.client.0.smithi046.stderr:5 2024-09-18T15:17:54.304 INFO:tasks.workunit.client.0.smithi046.stderr:9 2024-09-18T15:17:54.304 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:17:54.305 INFO:tasks.workunit.client.0.smithi046.stderr:6 2024-09-18T15:17:54.305 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:17:54.305 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:17:54.305 INFO:tasks.workunit.client.0.smithi046.stderr:7' 2024-09-18T15:17:54.305 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 6 2 1 7 2024-09-18T15:17:54.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local -a osds 2024-09-18T15:17:54.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-09-18T15:17:54.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-09-18T15:17:54.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-09-18T15:17:54.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:225: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-18T15:17:54.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local poolname=ecpool 2024-09-18T15:17:54.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local osd=3 2024-09-18T15:17:54.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:17:54.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:17:54.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:17:54.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-18T15:17:54.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:17:54.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:17:54.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:17:54.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:17:54.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-18T15:17:54.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:17:54.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-18T15:17:54.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:17:54.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:17:54.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:17:54.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:17:54.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:17:54.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:17:54.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:17:54.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:17:54.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:17:54.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-18T15:17:54.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:17:54.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:17:54.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-18T15:17:55.675 INFO:tasks.workunit.client.0.smithi046.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-09-18T15:17:56.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-18T15:17:56.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:17:56.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:17:56.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T15:17:56.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:17:56.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:17:56.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:17:56.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:17:56.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:17:56.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:17:56.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-18T15:17:56.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-18T15:17:56.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:17:56.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:17:56.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:17:56.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:17:56.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:17:56.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:17:56.404 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:17:56.404 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:17:56.404 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:17:56.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:17:56.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:17:56.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:17:56.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:17:56.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:17:56.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:17:56.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:17:56.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:17:56.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:17:56.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:17:56.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:17:56.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-18T15:17:56.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T15:17:56.407 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.3 2024-09-18T15:17:56.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:17:56.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-18T15:17:56.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T15:17:56.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:17:56.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:17:56.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:17:56.439 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:56.429+0000 7f6c7a2ad540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:56.442 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:56.433+0000 7f6c7a2ad540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:56.443 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:56.433+0000 7f6c7a2ad540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:56.445 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:56.437+0000 7f6c7a2ad540 -1 Falling back to public interface 2024-09-18T15:17:56.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T15:17:56.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:17:56.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:17:56.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:17:56.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:17:56.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:56.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:17:56.719 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:17:56.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:56.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:17:56.941 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:56.933+0000 7f6c7a2ad540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:17:56.956 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:56.949+0000 7f6c7a2ad540 -1 journal do_read_entry(491520): bad header magic 2024-09-18T15:17:56.957 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:56.949+0000 7f6c7a2ad540 -1 journal do_read_entry(491520): bad header magic 2024-09-18T15:17:56.964 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:17:56.957+0000 7f6c7a2ad540 -1 osd.3 67 log_to_monitors true 2024-09-18T15:17:57.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:58.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:58.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:58.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:17:58.507 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:17:58.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:58.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:17:59.210 INFO:tasks.workunit.client.0.smithi046.stderr:osd.3 up in weight 1 up_from 71 up_thru 71 down_at 68 last_clean_interval [20,67) [v2:127.0.0.1:6826/260005,v1:127.0.0.1:6827/260005] [v2:127.0.0.1:6828/260005,v1:127.0.0.1:6829/260005] exists,up 0fcca98d-f3b0-4b6a-8ec6-ccc43b7d6de2 2024-09-18T15:17:59.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:17:59.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:17:59.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:17:59.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:17:59.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:17:59.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:17:59.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:17:59.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:17:59.211 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:17:59.212 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:17:59.212 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:17:59.212 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:17:59.212 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:17:59.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:17:59.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:17:59.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:17:59.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:17:59.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:17:59.430 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:18:00.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:18:00.061 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:18:00.061 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:18:00.061 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:18:00.061 INFO:tasks.workunit.client.0.smithi046.stderr:4 2024-09-18T15:18:00.061 INFO:tasks.workunit.client.0.smithi046.stderr:5 2024-09-18T15:18:00.061 INFO:tasks.workunit.client.0.smithi046.stderr:6 2024-09-18T15:18:00.061 INFO:tasks.workunit.client.0.smithi046.stderr:7 2024-09-18T15:18:00.061 INFO:tasks.workunit.client.0.smithi046.stderr:8 2024-09-18T15:18:00.061 INFO:tasks.workunit.client.0.smithi046.stderr:9' 2024-09-18T15:18:00.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:18:00.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:00.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:18:00.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836499 2024-09-18T15:18:00.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836499 2024-09-18T15:18:00.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836499' 2024-09-18T15:18:00.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:00.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:18:00.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672979 2024-09-18T15:18:00.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672979 2024-09-18T15:18:00.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836499 1-42949672979' 2024-09-18T15:18:00.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:00.430 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:18:00.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509459 2024-09-18T15:18:00.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509459 2024-09-18T15:18:00.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836499 1-42949672979 2-64424509459' 2024-09-18T15:18:00.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:00.560 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:18:00.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=304942678018 2024-09-18T15:18:00.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 304942678018 2024-09-18T15:18:00.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836499 1-42949672979 2-64424509459 3-304942678018' 2024-09-18T15:18:00.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:00.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:18:00.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182417 2024-09-18T15:18:00.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182417 2024-09-18T15:18:00.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836499 1-42949672979 2-64424509459 3-304942678018 4-107374182417' 2024-09-18T15:18:00.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:00.844 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:18:01.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018895 2024-09-18T15:18:01.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018895 2024-09-18T15:18:01.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836499 1-42949672979 2-64424509459 3-304942678018 4-107374182417 5-128849018895' 2024-09-18T15:18:01.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:01.003 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:18:01.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855374 2024-09-18T15:18:01.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855374 2024-09-18T15:18:01.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836499 1-42949672979 2-64424509459 3-304942678018 4-107374182417 5-128849018895 6-150323855374' 2024-09-18T15:18:01.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:01.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:18:01.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691853 2024-09-18T15:18:01.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691853 2024-09-18T15:18:01.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836499 1-42949672979 2-64424509459 3-304942678018 4-107374182417 5-128849018895 6-150323855374 7-171798691853' 2024-09-18T15:18:01.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:01.283 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:18:01.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528331 2024-09-18T15:18:01.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528331 2024-09-18T15:18:01.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836499 1-42949672979 2-64424509459 3-304942678018 4-107374182417 5-128849018895 6-150323855374 7-171798691853 8-193273528331' 2024-09-18T15:18:01.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:01.423 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:18:01.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=214748364810 2024-09-18T15:18:01.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 214748364810 2024-09-18T15:18:01.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836499 1-42949672979 2-64424509459 3-304942678018 4-107374182417 5-128849018895 6-150323855374 7-171798691853 8-193273528331 9-214748364810' 2024-09-18T15:18:01.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:01.560 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836499 2024-09-18T15:18:01.560 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:01.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:18:01.564 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836499 2024-09-18T15:18:01.564 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:01.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836499 2024-09-18T15:18:01.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-09-18T15:18:01.566 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836499 2024-09-18T15:18:01.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:18:01.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836498 -lt 21474836499 2024-09-18T15:18:01.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:18:02.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:18:02.876 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:18:03.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836500 -lt 21474836499 2024-09-18T15:18:03.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:03.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672979 2024-09-18T15:18:03.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:03.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:18:03.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672979 2024-09-18T15:18:03.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:03.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672979 2024-09-18T15:18:03.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-18T15:18:03.210 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 42949672979 2024-09-18T15:18:03.211 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:18:03.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672979 -lt 42949672979 2024-09-18T15:18:03.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:03.552 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509459 2024-09-18T15:18:03.553 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:03.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:18:03.557 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509459 2024-09-18T15:18:03.557 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:03.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509459 2024-09-18T15:18:03.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-18T15:18:03.558 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509459 2024-09-18T15:18:03.559 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:18:03.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509459 -lt 64424509459 2024-09-18T15:18:03.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:03.875 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-304942678018 2024-09-18T15:18:03.875 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:03.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:18:03.878 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-304942678018 2024-09-18T15:18:03.878 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:03.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=304942678018 2024-09-18T15:18:03.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 304942678018' 2024-09-18T15:18:03.881 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.3 seq 304942678018 2024-09-18T15:18:03.881 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:18:04.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 304942678018 -lt 304942678018 2024-09-18T15:18:04.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:04.224 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182417 2024-09-18T15:18:04.224 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:04.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:18:04.227 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182417 2024-09-18T15:18:04.227 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:04.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182417 2024-09-18T15:18:04.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182417' 2024-09-18T15:18:04.230 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.4 seq 107374182417 2024-09-18T15:18:04.230 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:18:04.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182417 -lt 107374182417 2024-09-18T15:18:04.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:04.538 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018895 2024-09-18T15:18:04.538 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:04.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:18:04.541 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018895 2024-09-18T15:18:04.541 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:04.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018895 2024-09-18T15:18:04.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018895' 2024-09-18T15:18:04.543 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.5 seq 128849018895 2024-09-18T15:18:04.544 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:18:04.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018895 -lt 128849018895 2024-09-18T15:18:04.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:04.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855374 2024-09-18T15:18:04.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:04.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:18:04.872 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855374 2024-09-18T15:18:04.872 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:04.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855374 2024-09-18T15:18:04.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855374' 2024-09-18T15:18:04.875 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.6 seq 150323855374 2024-09-18T15:18:04.875 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:18:05.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855374 -lt 150323855374 2024-09-18T15:18:05.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:05.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691853 2024-09-18T15:18:05.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:05.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:18:05.277 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:05.277 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691853 2024-09-18T15:18:05.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691853 2024-09-18T15:18:05.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691853' 2024-09-18T15:18:05.278 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.7 seq 171798691853 2024-09-18T15:18:05.278 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:18:05.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691854 -lt 171798691853 2024-09-18T15:18:05.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:05.604 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528331 2024-09-18T15:18:05.604 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:05.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:18:05.608 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528331 2024-09-18T15:18:05.608 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:05.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528331 2024-09-18T15:18:05.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528331' 2024-09-18T15:18:05.610 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.8 seq 193273528331 2024-09-18T15:18:05.611 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:18:05.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528332 -lt 193273528331 2024-09-18T15:18:05.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:05.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-214748364810 2024-09-18T15:18:05.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:05.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:18:05.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-214748364810 2024-09-18T15:18:05.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:05.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=214748364810 2024-09-18T15:18:05.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 214748364810' 2024-09-18T15:18:05.987 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.9 seq 214748364810 2024-09-18T15:18:05.988 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:18:06.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 214748364811 -lt 214748364810 2024-09-18T15:18:06.323 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:18:06.323 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:18:06.324 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:18:06.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:18:06.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:18:06.694 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:18:06.694 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:18:06.694 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:18:06.694 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:18:06.694 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:18:06.694 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:18:07.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:18:07.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:18:07.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:18:07.051 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:18:07.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:18:07.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:18:07.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:18:07.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-18T15:18:07.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-09-18T15:18:07.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T15:18:07.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:18:07.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:18:07.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: local pg=2.0 2024-09-18T15:18:07.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: repair 2.0 2024-09-18T15:18:07.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-09-18T15:18:07.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-09-18T15:18:07.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:18:07.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:18:07.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:18:07.766 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:18:08.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T15:17:43.403042+0000 2024-09-18T15:18:08.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-09-18T15:18:08.423 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-18T15:18:08.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-09-18T15:17:43.403042+0000 2024-09-18T15:18:08.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=2.0 2024-09-18T15:18:08.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:17:43.403042+0000 2024-09-18T15:18:08.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:18:08.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:18:08.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:18:08.433 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:18:08.433 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:18:08.433 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:18:08.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:18:08.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:18:08.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:17:43.403042+0000 '>' 2024-09-18T15:17:43.403042+0000 2024-09-18T15:18:08.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:18:09.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:18:09.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:18:09.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:18:09.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:18:09.742 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:18:09.742 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:18:09.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:18:10.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:17:43.403042+0000 '>' 2024-09-18T15:17:43.403042+0000 2024-09-18T15:18:10.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:18:11.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:18:11.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:18:11.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:18:11.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:18:11.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:18:11.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:18:11.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:18:11.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:17:43.403042+0000 '>' 2024-09-18T15:17:43.403042+0000 2024-09-18T15:18:11.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:18:12.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:18:12.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:18:12.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:18:12.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:18:12.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:18:12.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:18:12.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:18:12.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:17:43.403042+0000 '>' 2024-09-18T15:17:43.403042+0000 2024-09-18T15:18:12.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:18:13.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:18:13.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:18:13.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:18:13.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:18:13.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:18:13.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:18:13.702 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:18:14.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:17:43.403042+0000 '>' 2024-09-18T15:17:43.403042+0000 2024-09-18T15:18:14.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:18:15.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:18:15.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:18:15.021 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:18:15.021 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:18:15.021 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:18:15.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:18:15.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:18:15.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:18:09.109169+0000 '>' 2024-09-18T15:17:43.403042+0000 2024-09-18T15:18:15.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:18:15.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:241: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-18T15:18:15.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:18:15.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:18:15.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-09-18T15:18:15.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:18:15.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-18T15:18:15.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:18:15.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:18:15.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-09-18T15:18:15.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:18:15.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-18T15:18:15.339 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:18:15.339 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:18:15.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:18:15.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:18:15.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:18:15.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:18:15.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-18T15:18:15.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:18:15.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:18:15.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-09-18T15:18:15.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:18:15.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-18T15:18:15.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-18T15:18:16.976 INFO:tasks.workunit.client.0.smithi046.stdout:_ 2024-09-18T15:18:16.976 INFO:tasks.workunit.client.0.smithi046.stdout:hinfo_key 2024-09-18T15:18:16.977 INFO:tasks.workunit.client.0.smithi046.stdout:snapset 2024-09-18T15:18:17.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-18T15:18:17.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:18:17.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:18:17.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-09-18T15:18:17.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:18:17.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-18T15:18:17.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:18:17.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:18:17.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:18:17.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:18:17.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-18T15:18:17.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-18T15:18:17.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:18:17.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:18:17.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:18:17.056 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:18:17.056 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:18:17.056 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:18:17.057 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:18:17.057 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:18:17.058 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:18:17.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:18:17.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:18:17.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:18:17.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:18:17.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:18:17.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:18:17.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:18:17.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:18:17.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:18:17.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:18:17.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:18:17.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-18T15:18:17.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-09-18T15:18:17.059 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.9 2024-09-18T15:18:17.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:18:17.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-18T15:18:17.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-09-18T15:18:17.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:18:17.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:18:17.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:18:17.092 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:18:17.080+0000 7f28d0475540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:18:17.095 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:18:17.084+0000 7f28d0475540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:18:17.097 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:18:17.088+0000 7f28d0475540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:18:17.099 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:18:17.088+0000 7f28d0475540 -1 Falling back to public interface 2024-09-18T15:18:17.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-09-18T15:18:17.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:18:17.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-09-18T15:18:17.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:18:17.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:18:17.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:18:17.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:18:17.506 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:18:17.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:18:17.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:18:17.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:18:18.558 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:18:18.548+0000 7f28d0475540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:18:18.621 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:18:18.612+0000 7f28d0475540 -1 journal do_read_entry(536576): bad header magic 2024-09-18T15:18:18.621 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:18:18.612+0000 7f28d0475540 -1 journal do_read_entry(536576): bad header magic 2024-09-18T15:18:18.635 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:18:18.624+0000 7f28d0475540 -1 osd.9 72 log_to_monitors true 2024-09-18T15:18:18.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:18:18.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:18:18.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:18:18.937 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:18:18.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:18:18.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:18:19.330 INFO:tasks.workunit.client.0.smithi046.stderr:osd.9 up in weight 1 up_from 77 up_thru 0 down_at 73 last_clean_interval [50,72) [v2:127.0.0.1:6874/261203,v1:127.0.0.1:6875/261203] [v2:127.0.0.1:6876/261203,v1:127.0.0.1:6877/261203] exists,up 57987690-7a54-4475-bda5-ffdec9be4750 2024-09-18T15:18:19.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:18:19.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:18:19.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:18:19.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:18:19.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:18:19.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:18:19.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:18:19.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:18:19.331 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:18:19.332 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:18:19.332 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:18:19.332 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:18:19.332 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:18:19.332 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:18:19.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:18:19.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:18:19.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:18:19.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:18:19.535 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:18:19.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:18:19.855 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:18:19.855 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:18:19.855 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:18:19.855 INFO:tasks.workunit.client.0.smithi046.stderr:4 2024-09-18T15:18:19.855 INFO:tasks.workunit.client.0.smithi046.stderr:5 2024-09-18T15:18:19.855 INFO:tasks.workunit.client.0.smithi046.stderr:6 2024-09-18T15:18:19.855 INFO:tasks.workunit.client.0.smithi046.stderr:7 2024-09-18T15:18:19.856 INFO:tasks.workunit.client.0.smithi046.stderr:8 2024-09-18T15:18:19.856 INFO:tasks.workunit.client.0.smithi046.stderr:9' 2024-09-18T15:18:19.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:18:19.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:19.856 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:18:20.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836504 2024-09-18T15:18:20.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836504 2024-09-18T15:18:20.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836504' 2024-09-18T15:18:20.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:20.011 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:18:20.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672984 2024-09-18T15:18:20.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672984 2024-09-18T15:18:20.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836504 1-42949672984' 2024-09-18T15:18:20.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:20.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:18:20.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509464 2024-09-18T15:18:20.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509464 2024-09-18T15:18:20.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836504 1-42949672984 2-64424509464' 2024-09-18T15:18:20.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:20.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:18:20.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=304942678023 2024-09-18T15:18:20.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 304942678023 2024-09-18T15:18:20.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836504 1-42949672984 2-64424509464 3-304942678023' 2024-09-18T15:18:20.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:20.634 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:18:20.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182422 2024-09-18T15:18:20.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182422 2024-09-18T15:18:20.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836504 1-42949672984 2-64424509464 3-304942678023 4-107374182422' 2024-09-18T15:18:20.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:20.781 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:18:20.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018900 2024-09-18T15:18:20.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018900 2024-09-18T15:18:20.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836504 1-42949672984 2-64424509464 3-304942678023 4-107374182422 5-128849018900' 2024-09-18T15:18:20.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:20.926 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:18:21.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855378 2024-09-18T15:18:21.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855378 2024-09-18T15:18:21.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836504 1-42949672984 2-64424509464 3-304942678023 4-107374182422 5-128849018900 6-150323855378' 2024-09-18T15:18:21.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:21.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:18:21.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691858 2024-09-18T15:18:21.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691858 2024-09-18T15:18:21.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836504 1-42949672984 2-64424509464 3-304942678023 4-107374182422 5-128849018900 6-150323855378 7-171798691858' 2024-09-18T15:18:21.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:21.214 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:18:21.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528336 2024-09-18T15:18:21.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528336 2024-09-18T15:18:21.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836504 1-42949672984 2-64424509464 3-304942678023 4-107374182422 5-128849018900 6-150323855378 7-171798691858 8-193273528336' 2024-09-18T15:18:21.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:21.347 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:18:21.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=330712481794 2024-09-18T15:18:21.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 330712481794 2024-09-18T15:18:21.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836504 1-42949672984 2-64424509464 3-304942678023 4-107374182422 5-128849018900 6-150323855378 7-171798691858 8-193273528336 9-330712481794' 2024-09-18T15:18:21.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:21.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836504 2024-09-18T15:18:21.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:21.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:18:21.484 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836504 2024-09-18T15:18:21.484 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:21.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836504 2024-09-18T15:18:21.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-18T15:18:21.486 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836504 2024-09-18T15:18:21.487 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:18:21.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836504 -lt 21474836504 2024-09-18T15:18:21.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:21.797 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672984 2024-09-18T15:18:21.797 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:21.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:18:21.801 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672984 2024-09-18T15:18:21.801 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:21.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672984 2024-09-18T15:18:21.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-09-18T15:18:21.803 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 42949672984 2024-09-18T15:18:21.803 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:18:22.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672983 -lt 42949672984 2024-09-18T15:18:22.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:18:23.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:18:23.135 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:18:23.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672984 -lt 42949672984 2024-09-18T15:18:23.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:23.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509464 2024-09-18T15:18:23.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:23.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:18:23.457 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509464 2024-09-18T15:18:23.457 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:23.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509464 2024-09-18T15:18:23.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-09-18T15:18:23.460 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509464 2024-09-18T15:18:23.460 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:18:24.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509464 -lt 64424509464 2024-09-18T15:18:24.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:24.091 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-304942678023 2024-09-18T15:18:24.091 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:24.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:18:24.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-304942678023 2024-09-18T15:18:24.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:24.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=304942678023 2024-09-18T15:18:24.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 304942678023' 2024-09-18T15:18:24.097 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.3 seq 304942678023 2024-09-18T15:18:24.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:18:24.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 304942678024 -lt 304942678023 2024-09-18T15:18:24.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:24.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182422 2024-09-18T15:18:24.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:24.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:18:24.416 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182422 2024-09-18T15:18:24.417 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:24.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182422 2024-09-18T15:18:24.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182422' 2024-09-18T15:18:24.419 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.4 seq 107374182422 2024-09-18T15:18:24.419 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:18:24.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182422 -lt 107374182422 2024-09-18T15:18:24.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:24.724 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018900 2024-09-18T15:18:24.724 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:24.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:18:24.727 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018900 2024-09-18T15:18:24.728 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:24.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018900 2024-09-18T15:18:24.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018900' 2024-09-18T15:18:24.730 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.5 seq 128849018900 2024-09-18T15:18:24.731 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:18:25.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018900 -lt 128849018900 2024-09-18T15:18:25.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:25.069 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855378 2024-09-18T15:18:25.069 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:25.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:18:25.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855378 2024-09-18T15:18:25.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:25.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855378 2024-09-18T15:18:25.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855378' 2024-09-18T15:18:25.075 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.6 seq 150323855378 2024-09-18T15:18:25.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:18:25.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855379 -lt 150323855378 2024-09-18T15:18:25.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:25.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691858 2024-09-18T15:18:25.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:25.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:18:25.418 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691858 2024-09-18T15:18:25.418 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:25.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691858 2024-09-18T15:18:25.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691858' 2024-09-18T15:18:25.420 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.7 seq 171798691858 2024-09-18T15:18:25.420 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:18:25.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691859 -lt 171798691858 2024-09-18T15:18:25.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:25.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528336 2024-09-18T15:18:25.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:25.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:18:25.755 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528336 2024-09-18T15:18:25.755 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:25.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528336 2024-09-18T15:18:25.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528336' 2024-09-18T15:18:25.757 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.8 seq 193273528336 2024-09-18T15:18:25.758 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:18:26.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528337 -lt 193273528336 2024-09-18T15:18:26.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:26.117 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-330712481794 2024-09-18T15:18:26.117 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:26.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:18:26.120 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-330712481794 2024-09-18T15:18:26.120 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:26.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=330712481794 2024-09-18T15:18:26.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 330712481794' 2024-09-18T15:18:26.122 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.9 seq 330712481794 2024-09-18T15:18:26.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:18:26.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 330712481795 -lt 330712481794 2024-09-18T15:18:26.467 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:18:26.468 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:18:26.468 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:18:26.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:18:26.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:18:26.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:18:26.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:18:26.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:18:26.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:18:26.886 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:18:26.886 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:18:27.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:18:27.223 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:18:27.224 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:18:27.224 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:18:27.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:18:27.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:18:27.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:18:27.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-18T15:18:27.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-18T15:18:27.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:260: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-09-18T15:18:27.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:225: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-18T15:18:27.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local poolname=ecpool 2024-09-18T15:18:27.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local osd=5 2024-09-18T15:18:27.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-18T15:18:27.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:18:27.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:18:27.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-09-18T15:18:27.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:18:27.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-18T15:18:27.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:18:27.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:18:27.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-09-18T15:18:27.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:18:27.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-18T15:18:27.654 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:18:27.654 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:18:27.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:18:27.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:18:27.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:18:27.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:18:27.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-18T15:18:27.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:18:27.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:18:27.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-09-18T15:18:27.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:18:27.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-18T15:18:27.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-18T15:18:29.960 INFO:tasks.workunit.client.0.smithi046.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-18T15:18:30.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-18T15:18:30.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:18:30.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:18:30.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-09-18T15:18:30.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:18:30.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-18T15:18:30.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:18:30.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:18:30.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:18:30.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:18:30.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-18T15:18:30.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-18T15:18:30.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:18:30.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:18:30.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:18:30.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:18:30.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:18:30.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:18:30.591 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:18:30.591 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:18:30.591 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:18:30.593 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:18:30.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:18:30.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:18:30.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:18:30.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:18:30.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:18:30.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:18:30.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:18:30.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:18:30.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:18:30.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:18:30.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-18T15:18:30.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-09-18T15:18:30.595 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.5 2024-09-18T15:18:30.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:18:30.596 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-18T15:18:30.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-09-18T15:18:30.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:18:30.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:18:30.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:18:30.628 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:18:30.616+0000 7feab94bf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:18:30.631 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:18:30.620+0000 7feab94bf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:18:30.633 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:18:30.624+0000 7feab94bf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:18:30.635 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:18:30.624+0000 7feab94bf540 -1 Falling back to public interface 2024-09-18T15:18:30.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-09-18T15:18:30.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:18:30.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T15:18:30.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:18:30.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:18:30.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:18:30.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:18:30.910 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:18:30.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:18:30.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:18:31.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:18:31.303 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:18:31.292+0000 7feab94bf540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:18:31.417 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:18:31.408+0000 7feab94bf540 -1 journal do_read_entry(708608): bad header magic 2024-09-18T15:18:31.417 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:18:31.408+0000 7feab94bf540 -1 journal do_read_entry(708608): bad header magic 2024-09-18T15:18:31.430 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:18:31.420+0000 7feab94bf540 -1 osd.5 78 log_to_monitors true 2024-09-18T15:18:32.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:18:32.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:18:32.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:18:32.242 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:18:32.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:18:32.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:18:32.689 INFO:tasks.workunit.client.0.smithi046.stderr:osd.5 up in weight 1 up_from 82 up_thru 82 down_at 79 last_clean_interval [30,78) [v2:127.0.0.1:6842/262193,v1:127.0.0.1:6843/262193] [v2:127.0.0.1:6844/262193,v1:127.0.0.1:6845/262193] exists,up 581d8192-9e17-497f-95a0-795d5338b5fe 2024-09-18T15:18:32.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:18:32.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:18:32.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:18:32.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:18:32.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:18:32.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:18:32.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:18:32.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:18:32.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:18:32.692 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:18:32.692 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:18:32.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:18:32.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:18:32.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:18:32.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:18:32.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:18:32.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:18:32.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:18:32.899 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:18:33.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:18:33.292 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:18:33.292 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:18:33.292 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:18:33.293 INFO:tasks.workunit.client.0.smithi046.stderr:4 2024-09-18T15:18:33.293 INFO:tasks.workunit.client.0.smithi046.stderr:5 2024-09-18T15:18:33.293 INFO:tasks.workunit.client.0.smithi046.stderr:6 2024-09-18T15:18:33.293 INFO:tasks.workunit.client.0.smithi046.stderr:7 2024-09-18T15:18:33.293 INFO:tasks.workunit.client.0.smithi046.stderr:8 2024-09-18T15:18:33.293 INFO:tasks.workunit.client.0.smithi046.stderr:9' 2024-09-18T15:18:33.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:18:33.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:33.293 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:18:33.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836508 2024-09-18T15:18:33.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836508 2024-09-18T15:18:33.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836508' 2024-09-18T15:18:33.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:33.429 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:18:33.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672987 2024-09-18T15:18:33.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672987 2024-09-18T15:18:33.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836508 1-42949672987' 2024-09-18T15:18:33.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:33.577 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:18:33.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509467 2024-09-18T15:18:33.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509467 2024-09-18T15:18:33.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836508 1-42949672987 2-64424509467' 2024-09-18T15:18:33.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:33.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:18:33.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=304942678027 2024-09-18T15:18:33.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 304942678027 2024-09-18T15:18:33.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836508 1-42949672987 2-64424509467 3-304942678027' 2024-09-18T15:18:33.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:33.843 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:18:33.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182425 2024-09-18T15:18:33.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182425 2024-09-18T15:18:33.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836508 1-42949672987 2-64424509467 3-304942678027 4-107374182425' 2024-09-18T15:18:33.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:33.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:18:34.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=352187318274 2024-09-18T15:18:34.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 352187318274 2024-09-18T15:18:34.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836508 1-42949672987 2-64424509467 3-304942678027 4-107374182425 5-352187318274' 2024-09-18T15:18:34.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:34.132 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:18:34.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855382 2024-09-18T15:18:34.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855382 2024-09-18T15:18:34.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836508 1-42949672987 2-64424509467 3-304942678027 4-107374182425 5-352187318274 6-150323855382' 2024-09-18T15:18:34.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:34.269 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:18:34.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691862 2024-09-18T15:18:34.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691862 2024-09-18T15:18:34.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836508 1-42949672987 2-64424509467 3-304942678027 4-107374182425 5-352187318274 6-150323855382 7-171798691862' 2024-09-18T15:18:34.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:34.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:18:34.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528340 2024-09-18T15:18:34.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528340 2024-09-18T15:18:34.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836508 1-42949672987 2-64424509467 3-304942678027 4-107374182425 5-352187318274 6-150323855382 7-171798691862 8-193273528340' 2024-09-18T15:18:34.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:34.544 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:18:34.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=330712481798 2024-09-18T15:18:34.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 330712481798 2024-09-18T15:18:34.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836508 1-42949672987 2-64424509467 3-304942678027 4-107374182425 5-352187318274 6-150323855382 7-171798691862 8-193273528340 9-330712481798' 2024-09-18T15:18:34.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:34.681 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836508 2024-09-18T15:18:34.681 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:34.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:18:34.684 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836508 2024-09-18T15:18:34.684 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:34.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836508 2024-09-18T15:18:34.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-09-18T15:18:34.686 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836508 2024-09-18T15:18:34.686 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:18:35.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836508 -lt 21474836508 2024-09-18T15:18:35.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:35.043 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672987 2024-09-18T15:18:35.043 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:35.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:18:35.047 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:35.047 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672987 2024-09-18T15:18:35.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672987 2024-09-18T15:18:35.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-09-18T15:18:35.048 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 42949672987 2024-09-18T15:18:35.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:18:35.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672988 -lt 42949672987 2024-09-18T15:18:35.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:35.350 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509467 2024-09-18T15:18:35.350 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:35.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:18:35.353 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509467 2024-09-18T15:18:35.353 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:35.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509467 2024-09-18T15:18:35.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-09-18T15:18:35.354 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509467 2024-09-18T15:18:35.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:18:35.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509468 -lt 64424509467 2024-09-18T15:18:35.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:35.661 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-304942678027 2024-09-18T15:18:35.662 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:35.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:18:35.665 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-304942678027 2024-09-18T15:18:35.665 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:35.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=304942678027 2024-09-18T15:18:35.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 304942678027' 2024-09-18T15:18:35.667 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.3 seq 304942678027 2024-09-18T15:18:35.668 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:18:36.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 304942678027 -lt 304942678027 2024-09-18T15:18:36.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:36.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182425 2024-09-18T15:18:36.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:36.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:18:36.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182425 2024-09-18T15:18:36.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:36.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182425 2024-09-18T15:18:36.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182425' 2024-09-18T15:18:36.024 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.4 seq 107374182425 2024-09-18T15:18:36.024 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:18:36.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182426 -lt 107374182425 2024-09-18T15:18:36.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:36.371 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-352187318274 2024-09-18T15:18:36.371 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:36.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:18:36.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-352187318274 2024-09-18T15:18:36.374 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:36.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=352187318274 2024-09-18T15:18:36.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 352187318274' 2024-09-18T15:18:36.376 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.5 seq 352187318274 2024-09-18T15:18:36.377 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:18:36.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 352187318274 -lt 352187318274 2024-09-18T15:18:36.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:36.808 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855382 2024-09-18T15:18:36.808 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:36.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:18:36.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855382 2024-09-18T15:18:36.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:36.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855382 2024-09-18T15:18:36.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855382' 2024-09-18T15:18:36.815 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.6 seq 150323855382 2024-09-18T15:18:36.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:18:37.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855383 -lt 150323855382 2024-09-18T15:18:37.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:37.150 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:37.150 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691862 2024-09-18T15:18:37.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:18:37.152 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691862 2024-09-18T15:18:37.152 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:37.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691862 2024-09-18T15:18:37.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691862' 2024-09-18T15:18:37.154 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.7 seq 171798691862 2024-09-18T15:18:37.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:18:37.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691862 -lt 171798691862 2024-09-18T15:18:37.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:37.464 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528340 2024-09-18T15:18:37.464 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:37.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:18:37.468 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528340 2024-09-18T15:18:37.468 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:37.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528340 2024-09-18T15:18:37.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528340' 2024-09-18T15:18:37.470 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.8 seq 193273528340 2024-09-18T15:18:37.470 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:18:37.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528340 -lt 193273528340 2024-09-18T15:18:37.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:37.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-330712481798 2024-09-18T15:18:37.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:37.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:18:37.809 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-330712481798 2024-09-18T15:18:37.809 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:37.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=330712481798 2024-09-18T15:18:37.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 330712481798' 2024-09-18T15:18:37.811 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.9 seq 330712481798 2024-09-18T15:18:37.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:18:38.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 330712481798 -lt 330712481798 2024-09-18T15:18:38.145 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:18:38.146 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:18:38.146 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:18:38.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:18:38.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:18:38.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:18:38.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:18:38.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:18:38.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:18:38.576 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:18:38.576 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:18:38.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:18:38.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:18:38.921 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:18:38.921 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:18:39.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:18:39.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:18:39.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:18:39.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-18T15:18:39.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-09-18T15:18:39.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T15:18:39.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:18:39.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:18:39.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: local pg=2.0 2024-09-18T15:18:39.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: repair 2.0 2024-09-18T15:18:39.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-09-18T15:18:39.719 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-09-18T15:18:39.719 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:18:39.719 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:18:39.719 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:18:39.719 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:18:40.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T15:18:09.109169+0000 2024-09-18T15:18:40.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-09-18T15:18:40.361 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-18T15:18:40.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-09-18T15:18:09.109169+0000 2024-09-18T15:18:40.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=2.0 2024-09-18T15:18:40.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:18:09.109169+0000 2024-09-18T15:18:40.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:18:40.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:18:40.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:18:40.371 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:18:40.371 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:18:40.371 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:18:40.371 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:18:40.372 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:18:40.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:18:09.109169+0000 '>' 2024-09-18T15:18:09.109169+0000 2024-09-18T15:18:40.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:18:41.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:18:41.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:18:41.677 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:18:41.677 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:18:41.677 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:18:41.677 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:18:41.678 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:18:41.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:18:09.109169+0000 '>' 2024-09-18T15:18:09.109169+0000 2024-09-18T15:18:41.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:18:42.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:18:42.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:18:42.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:18:42.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:18:42.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:18:42.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:18:42.998 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:18:43.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:18:09.109169+0000 '>' 2024-09-18T15:18:09.109169+0000 2024-09-18T15:18:43.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:18:44.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:18:44.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:18:44.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:18:44.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:18:44.310 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:18:44.310 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:18:44.310 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:18:44.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:18:09.109169+0000 '>' 2024-09-18T15:18:09.109169+0000 2024-09-18T15:18:44.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:18:45.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:18:45.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:18:45.626 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:18:45.626 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:18:45.626 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:18:45.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:18:45.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:18:45.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:18:09.109169+0000 '>' 2024-09-18T15:18:09.109169+0000 2024-09-18T15:18:45.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:18:46.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:18:46.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:18:46.943 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:18:46.943 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:18:46.944 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:18:46.944 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:18:46.944 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:18:47.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:18:09.109169+0000 '>' 2024-09-18T15:18:09.109169+0000 2024-09-18T15:18:47.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:18:48.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:18:48.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:18:48.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:18:48.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:18:48.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:18:48.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:18:48.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:18:48.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:18:46.272137+0000 '>' 2024-09-18T15:18:09.109169+0000 2024-09-18T15:18:48.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:18:48.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:241: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-18T15:18:48.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:18:48.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:18:48.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-09-18T15:18:48.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:18:48.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-18T15:18:48.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:18:48.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:18:48.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-09-18T15:18:48.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:18:48.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-18T15:18:48.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:18:48.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:18:48.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:18:48.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:18:48.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:18:48.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:18:48.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-18T15:18:48.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:18:48.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:18:48.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-09-18T15:18:48.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:18:48.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-18T15:18:48.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-18T15:18:50.933 INFO:tasks.workunit.client.0.smithi046.stdout:_ 2024-09-18T15:18:50.933 INFO:tasks.workunit.client.0.smithi046.stdout:hinfo_key 2024-09-18T15:18:50.933 INFO:tasks.workunit.client.0.smithi046.stdout:snapset 2024-09-18T15:18:51.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-18T15:18:51.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:18:51.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:18:51.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-09-18T15:18:51.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:18:51.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-18T15:18:51.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:18:51.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:18:51.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:18:51.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:18:51.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-18T15:18:51.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-18T15:18:51.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:18:51.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:18:51.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:18:51.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:18:51.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:18:51.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:18:51.437 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:18:51.438 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:18:51.438 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:18:51.438 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:18:51.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:18:51.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:18:51.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:18:51.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:18:51.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:18:51.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:18:51.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:18:51.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:18:51.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:18:51.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:18:51.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-18T15:18:51.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-09-18T15:18:51.439 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.9 2024-09-18T15:18:51.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:18:51.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-18T15:18:51.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-09-18T15:18:51.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:18:51.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:18:51.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:18:51.471 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:18:51.460+0000 7f9f42b5d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:18:51.475 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:18:51.464+0000 7f9f42b5d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:18:51.476 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:18:51.468+0000 7f9f42b5d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:18:51.480 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:18:51.468+0000 7f9f42b5d540 -1 Falling back to public interface 2024-09-18T15:18:51.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-09-18T15:18:51.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:18:51.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-09-18T15:18:51.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:18:51.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:18:51.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:18:51.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:18:51.736 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:18:51.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:18:51.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:18:52.145 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:18:52.136+0000 7f9f42b5d540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:18:52.151 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:18:52.140+0000 7f9f42b5d540 -1 journal do_read_entry(749568): bad header magic 2024-09-18T15:18:52.152 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:18:52.140+0000 7f9f42b5d540 -1 journal do_read_entry(749568): bad header magic 2024-09-18T15:18:52.165 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:18:52.156+0000 7f9f42b5d540 -1 osd.9 85 log_to_monitors true 2024-09-18T15:18:52.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:18:53.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:18:53.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:18:53.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:18:53.195 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:18:53.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:18:53.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:18:53.517 INFO:tasks.workunit.client.0.smithi046.stderr:osd.9 up in weight 1 up_from 88 up_thru 0 down_at 85 last_clean_interval [77,85) [v2:127.0.0.1:6874/263392,v1:127.0.0.1:6875/263392] [v2:127.0.0.1:6876/263392,v1:127.0.0.1:6877/263392] exists,up 57987690-7a54-4475-bda5-ffdec9be4750 2024-09-18T15:18:53.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:18:53.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:18:53.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:18:53.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:18:53.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:18:53.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:18:53.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:18:53.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:18:53.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:18:53.519 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:18:53.519 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:18:53.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:18:53.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:18:53.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:18:53.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:18:53.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:18:53.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:18:53.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:18:53.731 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:18:54.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:18:54.359 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:18:54.359 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:18:54.359 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:18:54.359 INFO:tasks.workunit.client.0.smithi046.stderr:4 2024-09-18T15:18:54.359 INFO:tasks.workunit.client.0.smithi046.stderr:5 2024-09-18T15:18:54.359 INFO:tasks.workunit.client.0.smithi046.stderr:6 2024-09-18T15:18:54.360 INFO:tasks.workunit.client.0.smithi046.stderr:7 2024-09-18T15:18:54.360 INFO:tasks.workunit.client.0.smithi046.stderr:8 2024-09-18T15:18:54.360 INFO:tasks.workunit.client.0.smithi046.stderr:9' 2024-09-18T15:18:54.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:18:54.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:54.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:18:54.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836513 2024-09-18T15:18:54.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836513 2024-09-18T15:18:54.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513' 2024-09-18T15:18:54.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:54.554 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:18:54.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672993 2024-09-18T15:18:54.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672993 2024-09-18T15:18:54.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513 1-42949672993' 2024-09-18T15:18:54.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:54.732 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:18:54.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509473 2024-09-18T15:18:54.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509473 2024-09-18T15:18:54.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-64424509473' 2024-09-18T15:18:54.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:54.871 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:18:55.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=304942678032 2024-09-18T15:18:55.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 304942678032 2024-09-18T15:18:55.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-64424509473 3-304942678032' 2024-09-18T15:18:55.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:55.329 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:18:55.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182431 2024-09-18T15:18:55.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182431 2024-09-18T15:18:55.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-64424509473 3-304942678032 4-107374182431' 2024-09-18T15:18:55.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:55.471 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:18:55.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=352187318279 2024-09-18T15:18:55.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 352187318279 2024-09-18T15:18:55.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-64424509473 3-304942678032 4-107374182431 5-352187318279' 2024-09-18T15:18:55.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:55.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:18:56.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855387 2024-09-18T15:18:56.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855387 2024-09-18T15:18:56.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-64424509473 3-304942678032 4-107374182431 5-352187318279 6-150323855387' 2024-09-18T15:18:56.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:56.037 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:18:56.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691867 2024-09-18T15:18:56.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691867 2024-09-18T15:18:56.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-64424509473 3-304942678032 4-107374182431 5-352187318279 6-150323855387 7-171798691867' 2024-09-18T15:18:56.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:56.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:18:56.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528345 2024-09-18T15:18:56.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528345 2024-09-18T15:18:56.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-64424509473 3-304942678032 4-107374182431 5-352187318279 6-150323855387 7-171798691867 8-193273528345' 2024-09-18T15:18:56.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:56.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:18:56.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=377957122050 2024-09-18T15:18:56.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 377957122050 2024-09-18T15:18:56.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-64424509473 3-304942678032 4-107374182431 5-352187318279 6-150323855387 7-171798691867 8-193273528345 9-377957122050' 2024-09-18T15:18:56.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:56.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836513 2024-09-18T15:18:56.498 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:56.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:18:56.501 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836513 2024-09-18T15:18:56.501 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:56.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836513 2024-09-18T15:18:56.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-09-18T15:18:56.502 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836513 2024-09-18T15:18:56.503 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:18:56.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836514 -lt 21474836513 2024-09-18T15:18:56.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:56.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672993 2024-09-18T15:18:56.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:56.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:18:56.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672993 2024-09-18T15:18:56.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:56.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672993 2024-09-18T15:18:56.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672993' 2024-09-18T15:18:56.822 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 42949672993 2024-09-18T15:18:56.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:18:57.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672993 -lt 42949672993 2024-09-18T15:18:57.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:57.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509473 2024-09-18T15:18:57.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:57.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:18:57.157 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509473 2024-09-18T15:18:57.157 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:57.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509473 2024-09-18T15:18:57.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509473' 2024-09-18T15:18:57.159 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509473 2024-09-18T15:18:57.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:18:57.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509473 -lt 64424509473 2024-09-18T15:18:57.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:57.587 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-304942678032 2024-09-18T15:18:57.588 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:57.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:18:57.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-304942678032 2024-09-18T15:18:57.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:57.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=304942678032 2024-09-18T15:18:57.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 304942678032' 2024-09-18T15:18:57.593 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.3 seq 304942678032 2024-09-18T15:18:57.594 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:18:57.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 304942678032 -lt 304942678032 2024-09-18T15:18:57.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:57.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182431 2024-09-18T15:18:57.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:57.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:18:57.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182431 2024-09-18T15:18:57.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:57.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182431 2024-09-18T15:18:57.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182431' 2024-09-18T15:18:57.933 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.4 seq 107374182431 2024-09-18T15:18:57.934 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:18:58.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182431 -lt 107374182431 2024-09-18T15:18:58.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:58.238 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-352187318279 2024-09-18T15:18:58.239 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:58.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:18:58.242 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-352187318279 2024-09-18T15:18:58.242 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:58.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=352187318279 2024-09-18T15:18:58.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 352187318279' 2024-09-18T15:18:58.244 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.5 seq 352187318279 2024-09-18T15:18:58.244 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:18:58.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 352187318280 -lt 352187318279 2024-09-18T15:18:58.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:58.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855387 2024-09-18T15:18:58.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:58.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:18:58.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855387 2024-09-18T15:18:58.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:58.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855387 2024-09-18T15:18:58.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855387' 2024-09-18T15:18:58.630 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.6 seq 150323855387 2024-09-18T15:18:58.631 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:18:58.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855388 -lt 150323855387 2024-09-18T15:18:58.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:58.986 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691867 2024-09-18T15:18:58.987 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:58.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:18:58.990 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691867 2024-09-18T15:18:58.990 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:58.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691867 2024-09-18T15:18:58.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691867' 2024-09-18T15:18:58.993 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.7 seq 171798691867 2024-09-18T15:18:58.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:18:59.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691868 -lt 171798691867 2024-09-18T15:18:59.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:59.330 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528345 2024-09-18T15:18:59.330 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:59.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:18:59.334 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:59.334 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528345 2024-09-18T15:18:59.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528345 2024-09-18T15:18:59.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528345' 2024-09-18T15:18:59.335 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.8 seq 193273528345 2024-09-18T15:18:59.336 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:18:59.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528346 -lt 193273528345 2024-09-18T15:18:59.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:59.723 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-377957122050 2024-09-18T15:18:59.723 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:59.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:18:59.727 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-377957122050 2024-09-18T15:18:59.727 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:59.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=377957122050 2024-09-18T15:18:59.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 377957122050' 2024-09-18T15:18:59.729 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.9 seq 377957122050 2024-09-18T15:18:59.729 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:19:00.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 377957122051 -lt 377957122050 2024-09-18T15:19:00.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:19:00.083 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:19:00.083 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:19:00.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:19:00.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:19:00.510 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:19:00.510 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:19:00.510 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:19:00.510 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:19:00.510 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:19:00.511 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:19:00.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:19:00.861 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:19:00.862 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:19:00.862 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:19:01.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:19:01.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:19:01.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:19:01.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-18T15:19:01.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-18T15:19:01.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-09-18T15:19:01.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:184: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-18T15:19:01.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local poolname=ecpool 2024-09-18T15:19:01.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local first=5 2024-09-18T15:19:01.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local second=9 2024-09-18T15:19:01.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:192: corrupt_and_repair_two: pids= 2024-09-18T15:19:01.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-18T15:19:01.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:19:01.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:19:01.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 264287"' 2024-09-18T15:19:01.305 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 264287' 2024-09-18T15:19:01.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-09-18T15:19:01.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:19:01.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:19:01.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-18T15:19:01.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 264289"' 2024-09-18T15:19:01.306 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 264289' 2024-09-18T15:19:01.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: wait_background pids 2024-09-18T15:19:01.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/264290: /' 2024-09-18T15:19:01.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: wait_background: pids=' 264287 264289' 2024-09-18T15:19:01.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: wait_background: return_code=0 2024-09-18T15:19:01.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:19:01.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 264287 2024-09-18T15:19:01.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-09-18T15:19:01.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/264293: /' 2024-09-18T15:19:04.508 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:19:04.509 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:19:04.512 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-09-18T15:19:04.512 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:19:04.512 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-09-18T15:19:04.512 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:19:04.683 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:19:04.683 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-09-18T15:19:04.683 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:19:04.683 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-18T15:19:04.684 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:19:04.684 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:19:04.684 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:19:04.684 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:19:04.685 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:19:04.685 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:19:04.685 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-09-18T15:19:04.685 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:19:04.685 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:19:04.685 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-09-18T15:19:04.685 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:19:04.686 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-18T15:19:04.686 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-09-18T15:19:04.686 INFO:tasks.workunit.client.0.smithi046.stderr:264293: remove 2#2:eb822e21:::SOMETHING:head# 2024-09-18T15:19:04.686 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-18T15:19:04.686 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:19:04.686 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:19:04.686 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-09-18T15:19:04.686 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:19:04.687 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-18T15:19:04.687 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:19:04.687 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:19:04.689 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:19:04.689 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:19:04.689 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-18T15:19:04.689 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-18T15:19:04.690 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:19:04.690 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:19:04.690 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:19:04.690 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:19:04.691 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:19:04.691 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:19:04.691 INFO:tasks.workunit.client.0.smithi046.stderr:264293: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:19:04.691 INFO:tasks.workunit.client.0.smithi046.stderr:264293: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:19:04.691 INFO:tasks.workunit.client.0.smithi046.stderr:264293: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:19:04.691 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:19:04.691 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:19:04.691 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:19:04.691 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:19:04.691 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:19:04.691 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:19:04.691 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:19:04.692 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:19:04.692 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:19:04.692 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:19:04.692 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:19:04.692 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-18T15:19:04.692 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-09-18T15:19:04.692 INFO:tasks.workunit.client.0.smithi046.stderr:264293: start osd.9 2024-09-18T15:19:04.692 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:19:04.692 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:19:04.692 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-09-18T15:19:04.692 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:19:04.692 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-18T15:19:04.693 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:19:04.693 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:19:04.693 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-09-18T15:19:04.693 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:19:04.693 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-18T15:19:04.693 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:19:04.693 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:19:04.693 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:19:04.693 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:19:04.694 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:19:04.694 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:19:04.694 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-18T15:19:04.694 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:19:04.694 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:19:04.694 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-09-18T15:19:04.694 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:19:04.694 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-18T15:19:04.694 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-18T15:19:04.694 INFO:tasks.workunit.client.0.smithi046.stderr:264290: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-18T15:19:04.694 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-18T15:19:04.694 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:19:04.694 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:19:04.695 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-09-18T15:19:04.695 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:19:04.695 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-18T15:19:04.695 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:19:04.695 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:19:04.695 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:19:04.695 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:19:04.695 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-18T15:19:04.695 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-18T15:19:04.695 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:19:04.695 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:19:04.696 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:19:04.696 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:19:04.696 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:19:04.696 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:19:04.696 INFO:tasks.workunit.client.0.smithi046.stderr:264290: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:19:04.696 INFO:tasks.workunit.client.0.smithi046.stderr:264290: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:19:04.696 INFO:tasks.workunit.client.0.smithi046.stderr:264290: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:19:04.696 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:19:04.696 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:19:04.696 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:19:04.696 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:19:04.696 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:19:04.696 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:19:04.697 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:19:04.697 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:19:04.697 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:19:04.697 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:19:04.697 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:19:04.697 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-18T15:19:04.697 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-09-18T15:19:04.697 INFO:tasks.workunit.client.0.smithi046.stderr:264290: start osd.5 2024-09-18T15:19:07.516 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:19:07.516 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-18T15:19:07.517 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-09-18T15:19:07.517 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:19:07.517 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:19:07.517 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:19:07.517 INFO:tasks.workunit.client.0.smithi046.stderr:264290: 2024-09-18T15:19:04.552+0000 7f416b72d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:07.517 INFO:tasks.workunit.client.0.smithi046.stderr:264290: 2024-09-18T15:19:04.556+0000 7f416b72d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:07.517 INFO:tasks.workunit.client.0.smithi046.stderr:264290: 2024-09-18T15:19:04.556+0000 7f416b72d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:07.517 INFO:tasks.workunit.client.0.smithi046.stderr:264290: 2024-09-18T15:19:04.568+0000 7f416b72d540 -1 Falling back to public interface 2024-09-18T15:19:07.517 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-09-18T15:19:07.517 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:19:07.517 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T15:19:07.518 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:19:07.518 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:19:07.518 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:07.518 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:19:07.518 INFO:tasks.workunit.client.0.smithi046.stderr:264290: 0 2024-09-18T15:19:07.518 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:07.518 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:19:07.518 INFO:tasks.workunit.client.0.smithi046.stderr:264290: 2024-09-18T15:19:05.316+0000 7f416b72d540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:19:07.518 INFO:tasks.workunit.client.0.smithi046.stderr:264290: 2024-09-18T15:19:05.324+0000 7f416b72d540 -1 journal do_read_entry(942080): bad header magic 2024-09-18T15:19:07.518 INFO:tasks.workunit.client.0.smithi046.stderr:264290: 2024-09-18T15:19:05.324+0000 7f416b72d540 -1 journal do_read_entry(942080): bad header magic 2024-09-18T15:19:07.518 INFO:tasks.workunit.client.0.smithi046.stderr:264290: 2024-09-18T15:19:05.332+0000 7f416b72d540 -1 osd.5 90 log_to_monitors true 2024-09-18T15:19:07.518 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:07.518 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:07.519 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:07.519 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:19:07.519 INFO:tasks.workunit.client.0.smithi046.stderr:264290: 1 2024-09-18T15:19:07.519 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:07.519 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:19:07.519 INFO:tasks.workunit.client.0.smithi046.stderr:264290: osd.5 up in weight 1 up_from 95 up_thru 95 down_at 91 last_clean_interval [82,90) [v2:127.0.0.1:6842/264443,v1:127.0.0.1:6844/264443] [v2:127.0.0.1:6846/264443,v1:127.0.0.1:6848/264443] exists,up 581d8192-9e17-497f-95a0-795d5338b5fe 2024-09-18T15:19:07.519 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:19:07.519 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:19:07.519 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:19:07.519 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:19:07.519 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:19:07.519 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:19:07.521 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_cleops 2024-09-18T15:19:07.521 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-18T15:19:07.521 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-09-18T15:19:07.521 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:19:07.521 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:19:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:19:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:264293: 2024-09-18T15:19:04.548+0000 7f7a3cd94540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:264293: 2024-09-18T15:19:04.552+0000 7f7a3cd94540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:264293: 2024-09-18T15:19:04.556+0000 7f7a3cd94540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:264293: 2024-09-18T15:19:04.568+0000 7f7a3cd94540 -1 Falling back to public interface 2024-09-18T15:19:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-09-18T15:19:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:19:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-09-18T15:19:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:19:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:19:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:07.522 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:19:07.523 INFO:tasks.workunit.client.0.smithi046.stderr:264293: 0 2024-09-18T15:19:07.523 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:07.523 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:19:07.523 INFO:tasks.workunit.client.0.smithi046.stderr:264293: 2024-09-18T15:19:05.300+0000 7f7a3cd94540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:19:07.523 INFO:tasks.workunit.client.0.smithi046.stderr:264293: 2024-09-18T15:19:05.304+0000 7f7a3cd94540 -1 journal do_read_entry(860160): bad header magic 2024-09-18T15:19:07.523 INFO:tasks.workunit.client.0.smithi046.stderr:264293: 2024-09-18T15:19:05.304+0000 7f7a3cd94540 -1 journal do_read_entry(860160): bad header magic 2024-09-18T15:19:07.523 INFO:tasks.workunit.client.0.smithi046.stderr:264293: 2024-09-18T15:19:05.316+0000 7f7a3cd94540 -1 osd.9 90 log_to_monitors true 2024-09-18T15:19:07.523 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:07.523 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:07.523 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:07.523 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:19:07.523 INFO:tasks.workunit.client.0.smithi046.stderr:264293: 1 2024-09-18T15:19:07.523 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:07.523 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:19:07.523 INFO:tasks.workunit.client.0.smithi046.stderr:264293: osd.9 up in weight 1 up_from 95 up_thru 0 down_at 91 last_clean_interval [88,90) [v2:127.0.0.1:6843/264429,v1:127.0.0.1:6845/264429] [v2:127.0.0.1:6847/264429,v1:127.0.0.1:6849/264429] exists,up 57987690-7a54-4475-bda5-ffdec9be4750 2024-09-18T15:19:07.524 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:19:07.524 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:19:07.524 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:19:07.524 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:19:07.524 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:19:07.524 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:19:09.283 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_cleaan: local cur_active_clean 2024-09-18T15:19:09.283 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:19:09.283 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:19:09.283 INFO:tasks.workunit.client.0.smithi046.stderr:264290: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:19:09.283 INFO:tasks.workunit.client.0.smithi046.stderr:264290: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:19:09.283 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:19:09.283 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:19:09.283 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:19:09.284 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:19:09.284 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:19:09.284 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:19:09.284 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:19:09.284 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:19:09.284 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:19:09.284 INFO:tasks.workunit.client.0.smithi046.stderr:264290: 1 2024-09-18T15:19:09.284 INFO:tasks.workunit.client.0.smithi046.stderr:264290: 2 2024-09-18T15:19:09.284 INFO:tasks.workunit.client.0.smithi046.stderr:264290: 3 2024-09-18T15:19:09.284 INFO:tasks.workunit.client.0.smithi046.stderr:264290: 4 2024-09-18T15:19:09.284 INFO:tasks.workunit.client.0.smithi046.stderr:264290: 5 2024-09-18T15:19:09.284 INFO:tasks.workunit.client.0.smithi046.stderr:264290: 6 2024-09-18T15:19:09.284 INFO:tasks.workunit.client.0.smithi046.stderr:264290: 7 2024-09-18T15:19:09.285 INFO:tasks.workunit.client.0.smithi046.stderr:264290: 8 2024-09-18T15:19:09.285 INFO:tasks.workunit.client.0.smithi046.stderr:264290: 9' 2024-09-18T15:19:09.285 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:19:09.285 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:09.285 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:19:09.285 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836518 2024-09-18T15:19:09.285 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836518 2024-09-18T15:19:09.285 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836518' 2024-09-18T15:19:09.285 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:09.285 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:19:09.285 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672997 2024-09-18T15:19:09.285 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672997 2024-09-18T15:19:09.285 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836518 1-42949672997' 2024-09-18T15:19:09.286 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:09.286 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:19:09.286 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509477 2024-09-18T15:19:09.286 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509477 2024-09-18T15:19:09.286 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-64424509477' 2024-09-18T15:19:09.286 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:09.286 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:19:09.286 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=304942678037 2024-09-18T15:19:09.286 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 304942678037 2024-09-18T15:19:09.286 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephn: local cur_active_clean 2024-09-18T15:19:09.287 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:19:09.287 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:19:09.287 INFO:tasks.workunit.client.0.smithi046.stderr:264293: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:19:09.287 INFO:tasks.workunit.client.0.smithi046.stderr:264293: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:19:09.287 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:19:09.287 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:19:09.287 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:19:09.287 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:19:09.287 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:19:09.287 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:19:09.287 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:19:09.287 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:19:09.287 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:19:09.288 INFO:tasks.workunit.client.0.smithi046.stderr:264293: 1 2024-09-18T15:19:09.288 INFO:tasks.workunit.client.0.smithi046.stderr:264293: 2 2024-09-18T15:19:09.288 INFO:tasks.workunit.client.0.smithi046.stderr:264293: 3 2024-09-18T15:19:09.288 INFO:tasks.workunit.client.0.smithi046.stderr:264293: 4 2024-09-18T15:19:09.288 INFO:tasks.workunit.client.0.smithi046.stderr:264293: 5 2024-09-18T15:19:09.288 INFO:tasks.workunit.client.0.smithi046.stderr:264293: 6 2024-09-18T15:19:09.288 INFO:tasks.workunit.client.0.smithi046.stderr:264293: 7 2024-09-18T15:19:09.288 INFO:tasks.workunit.client.0.smithi046.stderr:264293: 8 2024-09-18T15:19:09.288 INFO:tasks.workunit.client.0.smithi046.stderr:264293: 9' 2024-09-18T15:19:09.288 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:19:09.288 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:09.288 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:19:09.288 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836517 2024-09-18T15:19:09.288 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836517 2024-09-18T15:19:09.289 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836517' 2024-09-18T15:19:09.289 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:09.289 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:19:09.289 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672996 2024-09-18T15:19:09.289 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672996 2024-09-18T15:19:09.289 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836517 1-42949672996' 2024-09-18T15:19:09.289 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:09.289 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:19:09.289 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509478 2024-09-18T15:19:09.289 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509478 2024-09-18T15:19:09.289 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836517 1-42949672996 2-64424509478' 2024-09-18T15:19:09.289 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:09.289 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:19:09.289 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=304942678036 2024-09-18T15:19:09.290 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 304942678036 2024-09-18T15:19:10.093 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836517 1-42949672996 2-64424509478 3-304942678036' 2024-09-18T15:19:10.093 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:10.093 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:19:10.093 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182434 2024-09-18T15:19:10.093 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182434 2024-09-18T15:19:10.093 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836517 1-42949672996 2-64424509478 3-304942678036 4-107374182434' 2024-09-18T15:19:10.093 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:10.093 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:19:10.093 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=408021893123 2024-09-18T15:19:10.093 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 408021893123 2024-09-18T15:19:10.093 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836517 1-42949672996 2-64424509478 3-304942678036 4-107374182434 5-408021893123' 2024-09-18T15:19:10.094 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:10.094 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:19:10.094 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855391 2024-09-18T15:19:10.094 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855391 2024-09-18T15:19:10.094 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836517 1-42949672996 2-64424509478 3-304942678036 4-107374182434 5-408021893123 6-150323855391' 2024-09-18T15:19:10.094 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:10.094 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:19:10.094 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691871 2024-09-18T15:19:10.094 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691871 2024-09-18T15:19:10.094 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836517 1-42949672996 2-64424509478 3-304942678036 4-107374182434 5-408021893123 6-150323855391 7-171798691871' 2024-09-18T15:19:10.094 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:10.094 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:19:10.095 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528350 2024-09-18T15:19:10.095 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528350 2024-09-18T15:19:10.095 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836517 1-42949672996 2-64424509478 3-304942678036 4-107374182434 5-408021893123 6-150323855391 7-171798691871 8-193273528350' 2024-09-18T15:19:10.095 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:10.095 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:19:10.095 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=408021893122 2024-09-18T15:19:10.095 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 408021893122 2024-09-18T15:19:10.095 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/statest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-64424509477 3-304942678037' 2024-09-18T15:19:10.095 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:10.095 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:19:10.095 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182435 2024-09-18T15:19:10.095 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182435 2024-09-18T15:19:10.095 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-64424509477 3-304942678037 4-107374182435' 2024-09-18T15:19:10.096 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:10.096 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:19:10.096 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=408021893122 2024-09-18T15:19:10.096 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 408021893122 2024-09-18T15:19:10.096 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-64424509477 3-304942678037 4-107374182435 5-408021893122' 2024-09-18T15:19:10.096 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:10.096 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:19:10.096 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855392 2024-09-18T15:19:10.096 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855392 2024-09-18T15:19:10.096 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-64424509477 3-304942678037 4-107374182435 5-408021893122 6-150323855392' 2024-09-18T15:19:10.096 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:10.096 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:19:10.096 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691872 2024-09-18T15:19:10.096 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691872 2024-09-18T15:19:10.097 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-64424509477 3-304942678037 4-107374182435 5-408021893122 6-150323855392 7-171798691872' 2024-09-18T15:19:10.097 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:10.097 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:19:10.097 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528349 2024-09-18T15:19:10.097 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528349 2024-09-18T15:19:10.097 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-64424509477 3-304942678037 4-107374182435 5-408021893122 6-150323855392 7-171798691872 8-193273528349' 2024-09-18T15:19:10.097 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:10.097 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:19:10.097 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=408021893123 2024-09-18T15:19:10.097 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 408021893123 2024-09-18T15:19:11.344 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/stndalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836517 1-42949672996 2-64424509478 3-304942678036 4-107374182434 5-408021893123 6-150323855391 7-171798691871 8-193273528350 9-408021893122' 2024-09-18T15:19:11.344 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:11.344 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836517 2024-09-18T15:19:11.345 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:11.345 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:19:11.345 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836517 2024-09-18T15:19:11.345 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:11.345 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836517 2024-09-18T15:19:11.345 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2024-09-18T15:19:11.345 INFO:tasks.workunit.client.0.smithi046.stderr:264293: waiting osd.0 seq 21474836517 2024-09-18T15:19:11.345 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:19:11.345 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836518 -lt 21474836517 2024-09-18T15:19:11.345 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:11.345 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672996 2024-09-18T15:19:11.345 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:11.345 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:19:11.346 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672996 2024-09-18T15:19:11.346 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:11.346 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672996 2024-09-18T15:19:11.346 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672996' 2024-09-18T15:19:11.346 INFO:tasks.workunit.client.0.smithi046.stderr:264293: waiting osd.1 seq 42949672996 2024-09-18T15:19:11.346 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:19:11.346 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672998 -lt 42949672996 2024-09-18T15:19:11.346 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:11.346 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509478 2024-09-18T15:19:11.346 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:11.346 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:19:11.346 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509478 2024-09-18T15:19:11.346 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:11.347 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509478 2024-09-18T15:19:11.347 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509478' 2024-09-18T15:19:11.347 INFO:tasks.workunit.client.0.smithi046.stderr:264293: waiting osd.2 seq 64424509478 2024-09-18T15:19:11.347 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:19:11.347 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509478 -lt 64424509478 2024-09-18T15:19:11.347 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:11.347 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-304942678036andalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-64424509477 3-304942678037 4-107374182435 5-408021893122 6-150323855392 7-171798691872 8-193273528349 9-408021893123' 2024-09-18T15:19:11.347 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:11.347 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836518 2024-09-18T15:19:11.347 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:11.347 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:19:11.347 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836518 2024-09-18T15:19:11.348 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:11.348 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836518 2024-09-18T15:19:11.348 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2024-09-18T15:19:11.348 INFO:tasks.workunit.client.0.smithi046.stderr:264290: waiting osd.0 seq 21474836518 2024-09-18T15:19:11.348 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:19:11.348 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836518 -lt 21474836518 2024-09-18T15:19:11.348 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:11.348 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672997 2024-09-18T15:19:11.348 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:11.348 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:19:11.348 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672997 2024-09-18T15:19:11.348 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:11.348 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672997 2024-09-18T15:19:11.349 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672997' 2024-09-18T15:19:11.349 INFO:tasks.workunit.client.0.smithi046.stderr:264290: waiting osd.1 seq 42949672997 2024-09-18T15:19:11.349 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:19:11.349 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672998 -lt 42949672997 2024-09-18T15:19:11.349 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:11.349 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509477 2024-09-18T15:19:11.349 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:11.349 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:19:11.349 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509477 2024-09-18T15:19:11.349 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:11.349 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509477 2024-09-18T15:19:11.349 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509477' 2024-09-18T15:19:11.349 INFO:tasks.workunit.client.0.smithi046.stderr:264290: waiting osd.2 seq 64424509477 2024-09-18T15:19:11.350 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:19:11.350 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509478 -lt 64424509477 2024-09-18T15:19:11.350 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:11.350 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:12.386 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-304942678037 2024-09-18T15:19:12.387 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:19:12.387 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-304942678037 2024-09-18T15:19:12.387 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:12.387 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=304942678037 2024-09-18T15:19:12.387 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 304942678037' 2024-09-18T15:19:12.387 INFO:tasks.workunit.client.0.smithi046.stderr:264290: waiting osd.3 seq 304942678037 2024-09-18T15:19:12.388 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:19:12.388 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 304942678037 -lt 304942678037 2024-09-18T15:19:12.388 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:12.388 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:12.388 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182435 2024-09-18T15:19:12.389 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:19:12.389 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182435 2024-09-18T15:19:12.389 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:12.389 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182435 2024-09-18T15:19:12.389 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182435' 2024-09-18T15:19:12.389 INFO:tasks.workunit.client.0.smithi046.stderr:264290: waiting osd.4 seq 107374182435 2024-09-18T15:19:12.390 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:19:12.390 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182436 -lt 107374182435 2024-09-18T15:19:12.390 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:12.390 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-408021893122 2024-09-18T15:19:12.390 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:12.390 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:19:12.390 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-408021893122 2024-09-18T15:19:12.390 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:12.390 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=408021893122 2024-09-18T15:19:12.391 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 408021893122' 2024-09-18T15:19:12.391 INFO:tasks.workunit.client.0.smithi046.stderr:264290: waiting osd.5 seq 408021893122 2024-09-18T15:19:12.391 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:19:12.391 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 408021893124 -lt 408021893122 2024-09-18T15:19:12.391 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:12.391 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:12.391 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855392 2024-09-18T15:19:12.391 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:19:12.391 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855392 2024-09-18T15:19:12.392 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.s 2024-09-18T15:19:12.392 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:12.392 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:19:12.392 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-304942678036 2024-09-18T15:19:12.392 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:12.392 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=304942678036 2024-09-18T15:19:12.392 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 304942678036' 2024-09-18T15:19:12.392 INFO:tasks.workunit.client.0.smithi046.stderr:264293: waiting osd.3 seq 304942678036 2024-09-18T15:19:12.393 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:19:12.393 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 304942678037 -lt 304942678036 2024-09-18T15:19:12.393 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:12.393 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:12.393 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182434 2024-09-18T15:19:12.393 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:19:12.393 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182434 2024-09-18T15:19:12.393 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:12.393 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182434 2024-09-18T15:19:12.393 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182434' 2024-09-18T15:19:12.393 INFO:tasks.workunit.client.0.smithi046.stderr:264293: waiting osd.4 seq 107374182434 2024-09-18T15:19:12.393 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:19:12.394 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182436 -lt 107374182434 2024-09-18T15:19:12.394 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:12.394 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-408021893123 2024-09-18T15:19:12.394 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:12.394 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:19:12.394 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:12.394 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-408021893123 2024-09-18T15:19:12.394 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=408021893123 2024-09-18T15:19:12.394 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 408021893123' 2024-09-18T15:19:12.394 INFO:tasks.workunit.client.0.smithi046.stderr:264293: waiting osd.5 seq 408021893123 2024-09-18T15:19:12.394 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:19:12.394 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 408021893124 -lt 408021893123 2024-09-18T15:19:12.394 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:12.395 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855391 2024-09-18T15:19:12.395 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:12.395 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:19:12.395 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855391 2024-09-18T15:19:13.500 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:13.500 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855391 2024-09-18T15:19:13.500 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855391' 2024-09-18T15:19:13.500 INFO:tasks.workunit.client.0.smithi046.stderr:264293: waiting osd.6 seq 150323855391 2024-09-18T15:19:13.500 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:19:13.501 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855393 -lt 150323855391 2024-09-18T15:19:13.501 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:13.501 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691871 2024-09-18T15:19:13.501 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:13.501 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:19:13.501 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691871 2024-09-18T15:19:13.501 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:13.501 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691871 2024-09-18T15:19:13.501 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691871' 2024-09-18T15:19:13.502 INFO:tasks.workunit.client.0.smithi046.stderr:264293: waiting osd.7 seq 171798691871 2024-09-18T15:19:13.502 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:19:13.502 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691873 -lt 171798691871 2024-09-18T15:19:13.502 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:13.502 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528350 2024-09-18T15:19:13.502 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:13.502 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:19:13.502 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528350 2024-09-18T15:19:13.502 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:13.502 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528350 2024-09-18T15:19:13.502 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528350' 2024-09-18T15:19:13.502 INFO:tasks.workunit.client.0.smithi046.stderr:264293: waiting osd.8 seq 193273528350 2024-09-18T15:19:13.502 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:19:13.503 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528351 -lt 193273528350 2024-09-18T15:19:13.503 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:13.503 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-408021893122 2024-09-18T15:19:13.503 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:13.503 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:19:13.503 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-408021893122 2024-09-18T15:19:13.503 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:13.503 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=408021893122 2024-09-18T15:19:13.503 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 408021893122' 2024-09-18T15:19:13.503 INFO:tasks.workunit.client.0.smithi046.stderr:264293: waiting osd.9 seq 408021893122 2024-09-18T15:19:13.517 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:13.518 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855392 2024-09-18T15:19:13.518 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855392' 2024-09-18T15:19:13.518 INFO:tasks.workunit.client.0.smithi046.stderr:264290: waiting osd.6 seq 150323855392 2024-09-18T15:19:13.518 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:19:13.518 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855393 -lt 150323855392 2024-09-18T15:19:13.518 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:13.518 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691872 2024-09-18T15:19:13.518 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:13.518 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:19:13.518 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691872 2024-09-18T15:19:13.518 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:13.518 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691872 2024-09-18T15:19:13.519 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691872' 2024-09-18T15:19:13.519 INFO:tasks.workunit.client.0.smithi046.stderr:264290: waiting osd.7 seq 171798691872 2024-09-18T15:19:13.519 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:19:13.519 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691873 -lt 171798691872 2024-09-18T15:19:13.519 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:13.519 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:13.519 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528349 2024-09-18T15:19:13.519 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:19:13.520 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528349 2024-09-18T15:19:13.520 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:13.520 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528349 2024-09-18T15:19:13.520 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528349' 2024-09-18T15:19:13.520 INFO:tasks.workunit.client.0.smithi046.stderr:264290: waiting osd.8 seq 193273528349 2024-09-18T15:19:13.520 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:19:13.520 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528351 -lt 193273528349 2024-09-18T15:19:13.520 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:13.520 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-408021893123 2024-09-18T15:19:13.521 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:13.521 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:19:13.521 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-408021893123 2024-09-18T15:19:13.521 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:13.521 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=408021893123 2024-09-18T15:19:13.521 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 408021893123' 2024-09-18T15:19:13.521 INFO:tasks.workunit.client.0.smithi046.stderr:264290: waiting osd.9 seq 408021893123 2024-09-18T15:19:15.051 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: fush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:19:15.051 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 408021893124 -lt 408021893122 2024-09-18T15:19:15.051 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:19:15.051 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:19:15.051 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:19:15.051 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:19:15.052 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:19:15.052 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:19:15.052 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:19:15.052 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:19:15.052 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:19:15.052 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:19:15.052 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:19:15.052 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:19:15.053 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:19:15.053 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:19:15.053 INFO:tasks.workunit.client.0.smithi046.stderr:264293: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:19:15.053 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:19:15.053 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:19:15.053 INFO:tasks.workunit.client.0.smithi046.stderr:264293: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:19:15.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:19:15.056 INFO:tasks.workunit.client.0.smithi046.stderr:lush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:19:15.056 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 408021893124 -lt 408021893123 2024-09-18T15:19:15.056 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:19:15.056 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:19:15.056 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:19:15.056 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:19:15.056 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:19:15.057 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:19:15.057 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:19:15.057 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:19:15.057 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:19:15.057 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:19:15.057 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:19:15.057 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:19:15.057 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:19:15.057 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:19:15.058 INFO:tasks.workunit.client.0.smithi046.stderr:264290: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:19:15.058 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:19:15.058 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:19:15.058 INFO:tasks.workunit.client.0.smithi046.stderr:264290: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:19:15.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:19:15.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:19:15.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 264289 2024-09-18T15:19:15.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: eval 'pids='\'''\''' 2024-09-18T15:19:15.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: pids= 2024-09-18T15:19:15.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: wait_background: return 0 2024-09-18T15:19:15.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: return_code=0 2024-09-18T15:19:15.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-18T15:19:15.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-18T15:19:15.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-09-18T15:19:15.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T15:19:15.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:19:15.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:19:15.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: local pg=2.0 2024-09-18T15:19:15.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: repair 2.0 2024-09-18T15:19:15.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-09-18T15:19:15.403 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-09-18T15:19:15.403 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:19:15.403 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:19:15.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:19:15.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:19:15.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T15:18:46.272137+0000 2024-09-18T15:19:15.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-09-18T15:19:16.011 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-18T15:19:16.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-09-18T15:18:46.272137+0000 2024-09-18T15:19:16.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=2.0 2024-09-18T15:19:16.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:18:46.272137+0000 2024-09-18T15:19:16.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:19:16.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:19:16.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:19:16.024 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:19:16.024 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:19:16.024 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:19:16.024 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:19:16.025 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:19:16.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:18:46.272137+0000 '>' 2024-09-18T15:18:46.272137+0000 2024-09-18T15:19:16.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:19:17.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:19:17.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:19:17.334 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:19:17.334 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:19:17.334 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:19:17.334 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:19:17.335 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:19:17.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:18:46.272137+0000 '>' 2024-09-18T15:18:46.272137+0000 2024-09-18T15:19:17.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:19:18.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:19:18.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:19:18.649 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:19:18.649 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:19:18.649 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:19:18.650 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:19:18.650 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:19:18.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:18:46.272137+0000 '>' 2024-09-18T15:18:46.272137+0000 2024-09-18T15:19:18.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:19:19.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:19:19.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:19:19.967 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:19:19.967 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:19:19.967 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:19:19.967 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:19:19.967 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:19:20.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:18:46.272137+0000 '>' 2024-09-18T15:18:46.272137+0000 2024-09-18T15:19:20.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:19:21.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:19:21.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:19:21.278 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:19:21.278 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:19:21.278 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:19:21.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:19:21.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:19:21.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:18:46.272137+0000 '>' 2024-09-18T15:18:46.272137+0000 2024-09-18T15:19:21.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:19:22.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:19:22.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:19:22.593 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:19:22.593 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:19:22.593 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:19:22.593 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:19:22.594 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:19:22.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:18:46.272137+0000 '>' 2024-09-18T15:18:46.272137+0000 2024-09-18T15:19:22.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:19:23.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:19:23.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:19:23.907 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:19:23.907 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:19:23.907 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:19:23.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:19:23.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:19:24.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:18:46.272137+0000 '>' 2024-09-18T15:18:46.272137+0000 2024-09-18T15:19:24.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:19:25.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:19:25.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:19:25.223 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:19:25.223 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:19:25.224 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:19:25.224 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:19:25.224 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:19:25.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:19:18.125540+0000 '>' 2024-09-18T15:18:46.272137+0000 2024-09-18T15:19:25.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:19:25.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:207: corrupt_and_repair_two: pids= 2024-09-18T15:19:25.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-18T15:19:25.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:19:25.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:19:25.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 266463"' 2024-09-18T15:19:25.542 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 266463' 2024-09-18T15:19:25.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-18T15:19:25.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:19:25.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:19:25.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-18T15:19:25.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/266465: /' 2024-09-18T15:19:25.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 266466"' 2024-09-18T15:19:25.543 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 266466' 2024-09-18T15:19:25.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: wait_background pids 2024-09-18T15:19:25.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: wait_background: pids=' 266463 266466' 2024-09-18T15:19:25.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: wait_background: return_code=0 2024-09-18T15:19:25.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:19:25.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 266463 2024-09-18T15:19:25.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-18T15:19:25.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/266469: /' 2024-09-18T15:19:27.548 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:19:27.548 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:19:27.548 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-09-18T15:19:27.548 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:19:27.548 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-18T15:19:27.549 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:19:27.549 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:19:27.549 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-09-18T15:19:27.549 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:19:27.549 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-18T15:19:27.549 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:19:27.549 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:19:27.549 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:19:27.549 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:19:27.549 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:19:27.549 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:19:27.549 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-18T15:19:27.549 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:19:27.550 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:19:27.550 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-09-18T15:19:27.550 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:19:27.550 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-18T15:19:27.550 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-09-18T15:19:27.550 INFO:tasks.workunit.client.0.smithi046.stderr:266465: _ 2024-09-18T15:19:27.550 INFO:tasks.workunit.client.0.smithi046.stderr:266465: hinfo_key 2024-09-18T15:19:27.550 INFO:tasks.workunit.client.0.smithi046.stderr:266465: snapset 2024-09-18T15:19:27.550 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-18T15:19:27.550 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:19:27.550 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:19:27.550 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-09-18T15:19:27.550 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:19:27.550 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-18T15:19:27.550 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:19:27.551 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:19:27.554 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:19:27.555 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:19:27.555 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-18T15:19:27.555 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-18T15:19:27.555 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:19:27.555 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:19:27.555 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:19:27.555 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:19:27.555 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:19:27.555 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:19:27.555 INFO:tasks.workunit.client.0.smithi046.stderr:266465: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:19:27.555 INFO:tasks.workunit.client.0.smithi046.stderr:266465: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:19:27.555 INFO:tasks.workunit.client.0.smithi046.stderr:266465: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:19:27.556 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:19:27.556 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:19:27.556 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:19:27.556 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:19:27.556 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:19:27.556 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:19:27.556 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:19:27.556 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:19:27.556 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:19:27.556 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:19:27.556 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:19:27.556 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-18T15:19:27.556 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-09-18T15:19:27.556 INFO:tasks.workunit.client.0.smithi046.stderr:266465: start osd.5 2024-09-18T15:19:27.729 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:19:27.729 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:19:27.729 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-09-18T15:19:27.729 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:19:27.729 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-18T15:19:27.730 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:19:27.730 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:19:27.730 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-09-18T15:19:27.730 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:19:27.730 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-18T15:19:27.730 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:19:27.730 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:19:27.730 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:19:27.730 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:19:27.730 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:19:27.730 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:19:27.730 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-18T15:19:27.730 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:19:27.730 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:19:27.731 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-09-18T15:19:27.731 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:19:27.731 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-18T15:19:27.731 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-18T15:19:27.731 INFO:tasks.workunit.client.0.smithi046.stderr:266469: _ 2024-09-18T15:19:27.731 INFO:tasks.workunit.client.0.smithi046.stderr:266469: hinfo_key 2024-09-18T15:19:27.731 INFO:tasks.workunit.client.0.smithi046.stderr:266469: snapset 2024-09-18T15:19:27.731 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-18T15:19:27.731 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:19:27.731 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:19:27.731 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-09-18T15:19:27.731 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:19:27.731 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-18T15:19:27.732 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:19:27.732 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:19:27.735 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:19:27.735 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:19:27.735 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-18T15:19:27.735 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-18T15:19:27.736 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:19:27.736 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:19:27.736 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:19:27.736 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:19:27.736 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:19:27.736 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:19:27.736 INFO:tasks.workunit.client.0.smithi046.stderr:266469: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:19:27.736 INFO:tasks.workunit.client.0.smithi046.stderr:266469: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:19:27.736 INFO:tasks.workunit.client.0.smithi046.stderr:266469: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:19:27.736 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:19:27.736 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:19:27.737 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:19:27.737 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:19:27.737 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:19:27.737 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:19:27.737 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:19:27.737 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:19:27.737 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:19:27.737 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:19:27.737 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:19:27.737 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-18T15:19:27.737 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-09-18T15:19:27.737 INFO:tasks.workunit.client.0.smithi046.stderr:266469: start osd.9 2024-09-18T15:19:30.603 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:19:30.604 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-18T15:19:30.604 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-09-18T15:19:30.604 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:19:30.604 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:19:30.604 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:19:30.604 INFO:tasks.workunit.client.0.smithi046.stderr:266465: 2024-09-18T15:19:27.579+0000 7fa8041d5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:30.604 INFO:tasks.workunit.client.0.smithi046.stderr:266465: 2024-09-18T15:19:27.631+0000 7fa8041d5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:30.604 INFO:tasks.workunit.client.0.smithi046.stderr:266465: 2024-09-18T15:19:27.635+0000 7fa8041d5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:30.604 INFO:tasks.workunit.client.0.smithi046.stderr:266465: 2024-09-18T15:19:27.639+0000 7fa8041d5540 -1 Falling back to public interface 2024-09-18T15:19:30.604 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-09-18T15:19:30.605 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:19:30.605 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T15:19:30.605 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:19:30.605 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:19:30.605 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:30.605 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:19:30.605 INFO:tasks.workunit.client.0.smithi046.stderr:266465: 0 2024-09-18T15:19:30.605 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:30.605 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:19:30.605 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:30.605 INFO:tasks.workunit.client.0.smithi046.stderr:266465: 2024-09-18T15:19:28.815+0000 7fa8041d5540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:19:30.605 INFO:tasks.workunit.client.0.smithi046.stderr:266465: 2024-09-18T15:19:28.819+0000 7fa8041d5540 -1 journal do_read_entry(1085440): bad header magic 2024-09-18T15:19:30.605 INFO:tasks.workunit.client.0.smithi046.stderr:266465: 2024-09-18T15:19:28.819+0000 7fa8041d5540 -1 journal do_read_entry(1085440): bad header magic 2024-09-18T15:19:30.606 INFO:tasks.workunit.client.0.smithi046.stderr:266465: 2024-09-18T15:19:28.835+0000 7fa8041d5540 -1 osd.5 97 log_to_monitors true 2024-09-18T15:19:30.606 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:30.606 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:30.606 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:19:30.606 INFO:tasks.workunit.client.0.smithi046.stderr:266465: 1 2024-09-18T15:19:30.606 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:30.606 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:19:30.606 INFO:tasks.workunit.client.0.smithi046.stderr:266465: osd.5 up in weight 1 up_from 102 up_thru 102 down_at 98 last_clean_interval [95,97) [v2:127.0.0.1:6842/266597,v1:127.0.0.1:6843/266597] [v2:127.0.0.1:6844/266597,v1:127.0.0.1:6845/266597] exists,up 581d8192-9e17-497f-95a0-795d5338b5fe 2024-09-18T15:19:30.606 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:19:30.606 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:19:30.606 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:19:30.606 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:19:30.606 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:19:30.606 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:19:30.607 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644_recovery_ops 2024-09-18T15:19:30.607 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-18T15:19:30.607 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-09-18T15:19:30.607 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:19:30.607 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:19:30.607 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:19:30.607 INFO:tasks.workunit.client.0.smithi046.stderr:266469: 2024-09-18T15:19:27.767+0000 7f638bf0f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:30.607 INFO:tasks.workunit.client.0.smithi046.stderr:266469: 2024-09-18T15:19:28.211+0000 7f638bf0f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:30.607 INFO:tasks.workunit.client.0.smithi046.stderr:266469: 2024-09-18T15:19:28.211+0000 7f638bf0f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:30.608 INFO:tasks.workunit.client.0.smithi046.stderr:266469: 2024-09-18T15:19:28.215+0000 7f638bf0f540 -1 Falling back to public interface 2024-09-18T15:19:30.608 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-09-18T15:19:30.608 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:19:30.608 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-09-18T15:19:30.608 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:19:30.608 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:19:30.608 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:30.608 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:19:30.608 INFO:tasks.workunit.client.0.smithi046.stderr:266469: 0 2024-09-18T15:19:30.608 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:19:30.608 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:30.608 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:30.608 INFO:tasks.workunit.client.0.smithi046.stderr:266469: 2024-09-18T15:19:28.899+0000 7f638bf0f540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:19:30.609 INFO:tasks.workunit.client.0.smithi046.stderr:266469: 2024-09-18T15:19:28.907+0000 7f638bf0f540 -1 journal do_read_entry(983040): bad header magic 2024-09-18T15:19:30.609 INFO:tasks.workunit.client.0.smithi046.stderr:266469: 2024-09-18T15:19:28.907+0000 7f638bf0f540 -1 journal do_read_entry(983040): bad header magic 2024-09-18T15:19:30.609 INFO:tasks.workunit.client.0.smithi046.stderr:266469: 2024-09-18T15:19:28.915+0000 7f638bf0f540 -1 osd.9 97 log_to_monitors true 2024-09-18T15:19:30.609 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:30.609 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:30.609 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:19:30.609 INFO:tasks.workunit.client.0.smithi046.stderr:266469: 1 2024-09-18T15:19:30.609 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:30.609 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:19:30.609 INFO:tasks.workunit.client.0.smithi046.stderr:266469: osd.9 up in weight 1 up_from 102 up_thru 0 down_at 98 last_clean_interval [95,97) [v2:127.0.0.1:6874/266687,v1:127.0.0.1:6875/266687] [v2:127.0.0.1:6876/266687,v1:127.0.0.1:6877/266687] exists,up 57987690-7a54-4475-bda5-ffdec9be4750 2024-09-18T15:19:30.609 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:19:30.609 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:19:30.609 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:19:30.610 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:19:30.610 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:19:30.610 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:19:31.778 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wa: wait_for_clean: local cur_active_clean 2024-09-18T15:19:31.779 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:19:31.779 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:19:31.779 INFO:tasks.workunit.client.0.smithi046.stderr:266465: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:19:31.779 INFO:tasks.workunit.client.0.smithi046.stderr:266465: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:19:31.779 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:19:31.779 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:19:31.779 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:19:31.779 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:19:31.779 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:19:31.779 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:19:31.779 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:19:31.780 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:19:31.780 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:19:31.780 INFO:tasks.workunit.client.0.smithi046.stderr:266465: 1 2024-09-18T15:19:31.780 INFO:tasks.workunit.client.0.smithi046.stderr:266465: 2 2024-09-18T15:19:31.780 INFO:tasks.workunit.client.0.smithi046.stderr:266465: 3 2024-09-18T15:19:31.780 INFO:tasks.workunit.client.0.smithi046.stderr:266465: 4 2024-09-18T15:19:31.780 INFO:tasks.workunit.client.0.smithi046.stderr:266465: 5 2024-09-18T15:19:31.780 INFO:tasks.workunit.client.0.smithi046.stderr:266465: 6 2024-09-18T15:19:31.780 INFO:tasks.workunit.client.0.smithi046.stderr:266465: 7 2024-09-18T15:19:31.780 INFO:tasks.workunit.client.0.smithi046.stderr:266465: 8 2024-09-18T15:19:31.780 INFO:tasks.workunit.client.0.smithi046.stderr:266465: 9' 2024-09-18T15:19:31.780 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:19:31.780 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:31.781 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:19:31.781 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836524 2024-09-18T15:19:31.781 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836524 2024-09-18T15:19:31.781 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524' 2024-09-18T15:19:31.781 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:31.781 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:19:31.781 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673003 2024-09-18T15:19:31.781 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673003 2024-09-18T15:19:31.781 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949673003' 2024-09-18T15:19:31.781 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:31.781 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:19:31.781 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509483 2024-09-18T15:19:31.781 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509483 2024-09-18T15:19:31.782 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949673003 2-64424509483' 2024-09-18T15:19:31.782 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:31.782 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:19:31.782 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=304942678042 2024-09-18T15:19:31.782 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 304942678042 2024-09-18T15:19:31.790 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /hoit_for_clean: local cur_active_clean 2024-09-18T15:19:31.791 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:19:31.791 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:19:31.791 INFO:tasks.workunit.client.0.smithi046.stderr:266469: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:19:31.791 INFO:tasks.workunit.client.0.smithi046.stderr:266469: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:19:31.791 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:19:31.791 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:19:31.791 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:19:31.791 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:19:31.791 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:19:31.791 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:19:31.791 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:19:31.791 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:19:31.791 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:19:31.792 INFO:tasks.workunit.client.0.smithi046.stderr:266469: 1 2024-09-18T15:19:31.792 INFO:tasks.workunit.client.0.smithi046.stderr:266469: 2 2024-09-18T15:19:31.792 INFO:tasks.workunit.client.0.smithi046.stderr:266469: 3 2024-09-18T15:19:31.792 INFO:tasks.workunit.client.0.smithi046.stderr:266469: 4 2024-09-18T15:19:31.792 INFO:tasks.workunit.client.0.smithi046.stderr:266469: 5 2024-09-18T15:19:31.792 INFO:tasks.workunit.client.0.smithi046.stderr:266469: 6 2024-09-18T15:19:31.792 INFO:tasks.workunit.client.0.smithi046.stderr:266469: 7 2024-09-18T15:19:31.792 INFO:tasks.workunit.client.0.smithi046.stderr:266469: 8 2024-09-18T15:19:31.792 INFO:tasks.workunit.client.0.smithi046.stderr:266469: 9' 2024-09-18T15:19:31.792 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:19:31.792 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:31.792 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:19:31.792 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836525 2024-09-18T15:19:31.793 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836525 2024-09-18T15:19:31.793 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836525' 2024-09-18T15:19:31.793 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:31.793 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:19:31.793 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673004 2024-09-18T15:19:31.793 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673004 2024-09-18T15:19:31.793 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836525 1-42949673004' 2024-09-18T15:19:31.793 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:31.793 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:19:31.793 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509484 2024-09-18T15:19:31.793 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509484 2024-09-18T15:19:31.793 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836525 1-42949673004 2-64424509484' 2024-09-18T15:19:31.793 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:31.793 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:19:31.794 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=304942678043 2024-09-18T15:19:31.794 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 304942678043 2024-09-18T15:19:32.572 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ume/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949673003 2-64424509483 3-304942678042' 2024-09-18T15:19:32.572 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:32.572 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:19:32.572 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182441 2024-09-18T15:19:32.572 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182441 2024-09-18T15:19:32.573 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949673003 2-64424509483 3-304942678042 4-107374182441' 2024-09-18T15:19:32.573 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:32.573 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:19:32.573 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=438086664194 2024-09-18T15:19:32.573 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 438086664194 2024-09-18T15:19:32.573 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949673003 2-64424509483 3-304942678042 4-107374182441 5-438086664194' 2024-09-18T15:19:32.573 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:32.573 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:19:32.573 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855398 2024-09-18T15:19:32.573 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855398 2024-09-18T15:19:32.573 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949673003 2-64424509483 3-304942678042 4-107374182441 5-438086664194 6-150323855398' 2024-09-18T15:19:32.573 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:32.573 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:19:32.574 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691879 2024-09-18T15:19:32.574 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691879 2024-09-18T15:19:32.574 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949673003 2-64424509483 3-304942678042 4-107374182441 5-438086664194 6-150323855398 7-171798691879' 2024-09-18T15:19:32.574 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:32.574 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:19:32.574 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528357 2024-09-18T15:19:32.574 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528357 2024-09-18T15:19:32.574 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949673003 2-64424509483 3-304942678042 4-107374182441 5-438086664194 6-150323855398 7-171798691879 8-193273528357' 2024-09-18T15:19:32.574 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:32.574 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:19:32.574 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=438086664194 2024-09-18T15:19:32.574 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 438086664194 2024-09-18T15:19:32.575 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.buntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836525 1-42949673004 2-64424509484 3-304942678043' 2024-09-18T15:19:32.575 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:32.575 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:19:32.575 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182442 2024-09-18T15:19:32.575 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182442 2024-09-18T15:19:32.575 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836525 1-42949673004 2-64424509484 3-304942678043 4-107374182442' 2024-09-18T15:19:32.575 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:32.575 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:19:32.575 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=438086664195 2024-09-18T15:19:32.575 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 438086664195 2024-09-18T15:19:32.575 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836525 1-42949673004 2-64424509484 3-304942678043 4-107374182442 5-438086664195' 2024-09-18T15:19:32.575 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:32.576 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:19:32.576 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855399 2024-09-18T15:19:32.576 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855399 2024-09-18T15:19:32.576 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836525 1-42949673004 2-64424509484 3-304942678043 4-107374182442 5-438086664195 6-150323855399' 2024-09-18T15:19:32.576 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:32.576 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:19:32.576 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691878 2024-09-18T15:19:32.576 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691878 2024-09-18T15:19:32.576 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836525 1-42949673004 2-64424509484 3-304942678043 4-107374182442 5-438086664195 6-150323855399 7-171798691878' 2024-09-18T15:19:32.576 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:32.576 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:19:32.576 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528356 2024-09-18T15:19:32.576 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528356 2024-09-18T15:19:32.577 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836525 1-42949673004 2-64424509484 3-304942678043 4-107374182442 5-438086664195 6-150323855399 7-171798691878 8-193273528356' 2024-09-18T15:19:32.577 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:32.577 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:19:32.577 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=438086664195 2024-09-18T15:19:32.577 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 438086664195 2024-09-18T15:19:33.576 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836525 1-42949673004 2-64424509484 3-304942678043 4-107374182442 5-438086664195 6-150323855399 7-171798691878 8-193273528356 9-438086664195' 2024-09-18T15:19:33.576 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:33.577 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836525 2024-09-18T15:19:33.577 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:33.577 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:19:33.577 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836525 2024-09-18T15:19:33.577 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:33.577 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836525 2024-09-18T15:19:33.577 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2024-09-18T15:19:33.577 INFO:tasks.workunit.client.0.smithi046.stderr:266469: waiting osd.0 seq 21474836525 2024-09-18T15:19:33.577 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:19:33.577 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836525 -lt 21474836525 2024-09-18T15:19:33.577 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:33.577 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673004 2024-09-18T15:19:33.577 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:33.578 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:19:33.578 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673004 2024-09-18T15:19:33.578 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:33.578 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673004 2024-09-18T15:19:33.578 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673004' 2024-09-18T15:19:33.578 INFO:tasks.workunit.client.0.smithi046.stderr:266469: waiting osd.1 seq 42949673004 2024-09-18T15:19:33.578 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:19:33.578 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673004 -lt 42949673004 2024-09-18T15:19:33.578 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:33.578 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:33.578 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509484 2024-09-18T15:19:33.578 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:19:33.578 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509484 2024-09-18T15:19:33.578 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:33.579 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509484 2024-09-18T15:19:33.579 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509484' 2024-09-18T15:19:33.579 INFO:tasks.workunit.client.0.smithi046.stderr:266469: waiting osd.2 seq 64424509484 2024-09-18T15:19:33.579 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:19:33.579 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509484 -lt 64424509484 2024-09-18T15:19:33.579 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:33.579 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d -client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949673003 2-64424509483 3-304942678042 4-107374182441 5-438086664194 6-150323855398 7-171798691879 8-193273528357 9-438086664194' 2024-09-18T15:19:33.579 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:33.579 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836524 2024-09-18T15:19:33.579 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:33.580 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:19:33.580 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836524 2024-09-18T15:19:33.580 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:33.580 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836524 2024-09-18T15:19:33.580 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-09-18T15:19:33.580 INFO:tasks.workunit.client.0.smithi046.stderr:266465: waiting osd.0 seq 21474836524 2024-09-18T15:19:33.580 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:19:33.580 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836525 -lt 21474836524 2024-09-18T15:19:33.580 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:33.580 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673003 2024-09-18T15:19:33.580 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:33.580 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:19:33.580 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673003 2024-09-18T15:19:33.581 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:33.581 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673003 2024-09-18T15:19:33.581 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673003' 2024-09-18T15:19:33.581 INFO:tasks.workunit.client.0.smithi046.stderr:266465: waiting osd.1 seq 42949673003 2024-09-18T15:19:33.581 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:19:33.581 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673004 -lt 42949673003 2024-09-18T15:19:33.581 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:33.581 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509483 2024-09-18T15:19:33.581 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:33.581 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:19:33.581 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509483 2024-09-18T15:19:33.581 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:33.581 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509483 2024-09-18T15:19:33.582 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509483' 2024-09-18T15:19:33.582 INFO:tasks.workunit.client.0.smithi046.stderr:266465: waiting osd.2 seq 64424509483 2024-09-18T15:19:33.582 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:19:33.582 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509484 -lt 64424509483 2024-09-18T15:19:33.582 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:34.669 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-304942678042 2024-09-18T15:19:34.670 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:34.670 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:19:34.670 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-304942678042 2024-09-18T15:19:34.670 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:34.670 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=304942678042 2024-09-18T15:19:34.670 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 304942678042' 2024-09-18T15:19:34.670 INFO:tasks.workunit.client.0.smithi046.stderr:266465: waiting osd.3 seq 304942678042 2024-09-18T15:19:34.670 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:19:34.670 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 304942678043 -lt 304942678042 2024-09-18T15:19:34.671 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:34.671 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182441 2024-09-18T15:19:34.671 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:34.671 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:19:34.671 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182441 2024-09-18T15:19:34.671 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:34.671 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182441 2024-09-18T15:19:34.671 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182441' 2024-09-18T15:19:34.671 INFO:tasks.workunit.client.0.smithi046.stderr:266465: waiting osd.4 seq 107374182441 2024-09-18T15:19:34.671 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:19:34.671 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182442 -lt 107374182441 2024-09-18T15:19:34.671 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:34.672 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-438086664194 2024-09-18T15:19:34.672 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:34.672 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:19:34.672 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:34.672 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-438086664194 2024-09-18T15:19:34.672 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=438086664194 2024-09-18T15:19:34.672 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 438086664194' 2024-09-18T15:19:34.672 INFO:tasks.workunit.client.0.smithi046.stderr:266465: waiting osd.5 seq 438086664194 2024-09-18T15:19:34.672 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:19:34.672 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 438086664196 -lt 438086664194 2024-09-18T15:19:34.672 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:34.672 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855398 2024-09-18T15:19:34.672 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:34.672 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:19:34.673 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855398 2024-09-18T15:19:34.673 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ -f 1 2024-09-18T15:19:34.673 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-304942678043 2024-09-18T15:19:34.673 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:19:34.673 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-304942678043 2024-09-18T15:19:34.673 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:34.673 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=304942678043 2024-09-18T15:19:34.673 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 304942678043' 2024-09-18T15:19:34.674 INFO:tasks.workunit.client.0.smithi046.stderr:266469: waiting osd.3 seq 304942678043 2024-09-18T15:19:34.674 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:19:34.674 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 304942678043 -lt 304942678043 2024-09-18T15:19:34.674 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:34.674 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182442 2024-09-18T15:19:34.674 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:34.674 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:19:34.674 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:34.674 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182442 2024-09-18T15:19:34.674 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182442 2024-09-18T15:19:34.674 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182442' 2024-09-18T15:19:34.674 INFO:tasks.workunit.client.0.smithi046.stderr:266469: waiting osd.4 seq 107374182442 2024-09-18T15:19:34.674 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:19:34.674 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182442 -lt 107374182442 2024-09-18T15:19:34.674 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:34.675 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-438086664195 2024-09-18T15:19:34.675 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:34.675 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:19:34.675 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-438086664195 2024-09-18T15:19:34.675 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:34.675 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=438086664195 2024-09-18T15:19:34.675 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 438086664195' 2024-09-18T15:19:34.675 INFO:tasks.workunit.client.0.smithi046.stderr:266469: waiting osd.5 seq 438086664195 2024-09-18T15:19:34.675 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:19:34.675 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 438086664196 -lt 438086664195 2024-09-18T15:19:34.675 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:34.675 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855399 2024-09-18T15:19:34.675 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:34.675 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:19:34.676 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855399 2024-09-18T15:19:35.685 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:35.686 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855399 2024-09-18T15:19:35.686 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855399' 2024-09-18T15:19:35.686 INFO:tasks.workunit.client.0.smithi046.stderr:266469: waiting osd.6 seq 150323855399 2024-09-18T15:19:35.686 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:19:35.686 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855399 -lt 150323855399 2024-09-18T15:19:35.686 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:35.686 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691878 2024-09-18T15:19:35.686 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:35.686 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:19:35.686 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691878 2024-09-18T15:19:35.686 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:35.686 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691878 2024-09-18T15:19:35.687 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691878' 2024-09-18T15:19:35.687 INFO:tasks.workunit.client.0.smithi046.stderr:266469: waiting osd.7 seq 171798691878 2024-09-18T15:19:35.687 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:19:35.687 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691879 -lt 171798691878 2024-09-18T15:19:35.687 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:35.687 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528356 2024-09-18T15:19:35.687 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:35.687 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:19:35.687 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528356 2024-09-18T15:19:35.687 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:35.687 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528356 2024-09-18T15:19:35.687 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528356' 2024-09-18T15:19:35.687 INFO:tasks.workunit.client.0.smithi046.stderr:266469: waiting osd.8 seq 193273528356 2024-09-18T15:19:35.688 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:19:35.688 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528357 -lt 193273528356 2024-09-18T15:19:35.688 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:35.688 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-438086664195 2024-09-18T15:19:35.688 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:35.688 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:19:35.688 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-438086664195 2024-09-18T15:19:35.688 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:35.688 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=438086664195 2024-09-18T15:19:35.688 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 438086664195' 2024-09-18T15:19:35.688 INFO:tasks.workunit.client.0.smithi046.stderr:266469: waiting osd.9 seq 438086664195 2024-09-18T15:19:35.706 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:35.706 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855398 2024-09-18T15:19:35.706 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855398' 2024-09-18T15:19:35.706 INFO:tasks.workunit.client.0.smithi046.stderr:266465: waiting osd.6 seq 150323855398 2024-09-18T15:19:35.706 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:19:35.706 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855399 -lt 150323855398 2024-09-18T15:19:35.706 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:35.707 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691879 2024-09-18T15:19:35.707 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:35.707 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:19:35.707 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691879 2024-09-18T15:19:35.707 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:35.707 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691879 2024-09-18T15:19:35.707 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691879' 2024-09-18T15:19:35.707 INFO:tasks.workunit.client.0.smithi046.stderr:266465: waiting osd.7 seq 171798691879 2024-09-18T15:19:35.707 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:19:35.707 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691879 -lt 171798691879 2024-09-18T15:19:35.707 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:35.707 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528357 2024-09-18T15:19:35.707 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:35.707 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:19:35.707 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528357 2024-09-18T15:19:35.708 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:35.708 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528357 2024-09-18T15:19:35.708 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528357' 2024-09-18T15:19:35.708 INFO:tasks.workunit.client.0.smithi046.stderr:266465: waiting osd.8 seq 193273528357 2024-09-18T15:19:35.708 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:19:35.708 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528357 -lt 193273528357 2024-09-18T15:19:35.708 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:35.708 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:35.708 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-438086664194 2024-09-18T15:19:35.708 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:19:35.708 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-438086664194 2024-09-18T15:19:35.708 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:35.708 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=438086664194 2024-09-18T15:19:35.708 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 438086664194' 2024-09-18T15:19:35.709 INFO:tasks.workunit.client.0.smithi046.stderr:266465: waiting osd.9 seq 438086664194 2024-09-18T15:19:37.093 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpsh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:19:37.094 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 438086664196 -lt 438086664195 2024-09-18T15:19:37.094 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:19:37.094 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:19:37.094 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:19:37.094 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:19:37.094 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:19:37.094 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:19:37.094 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:19:37.094 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:19:37.095 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:19:37.095 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:19:37.095 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:19:37.095 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:19:37.095 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:19:37.095 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:19:37.095 INFO:tasks.workunit.client.0.smithi046.stderr:266469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:19:37.095 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:19:37.095 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:19:37.095 INFO:tasks.workunit.client.0.smithi046.stderr:266469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:19:37.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:19:37.154 INFO:tasks.workunit.client.0.smithi046.stderr:ers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:19:37.154 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 438086664196 -lt 438086664194 2024-09-18T15:19:37.155 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:19:37.155 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:19:37.155 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:19:37.155 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:19:37.155 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:19:37.155 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:19:37.155 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:19:37.155 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:19:37.155 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:19:37.155 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:19:37.155 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:19:37.155 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:19:37.155 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:19:37.156 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:19:37.156 INFO:tasks.workunit.client.0.smithi046.stderr:266465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:19:37.156 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:19:37.156 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:19:37.156 INFO:tasks.workunit.client.0.smithi046.stderr:266465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:19:37.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:19:37.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:19:37.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 266466 2024-09-18T15:19:37.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: eval 'pids='\'''\''' 2024-09-18T15:19:37.156 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: pids= 2024-09-18T15:19:37.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: wait_background: return 0 2024-09-18T15:19:37.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: return_code=0 2024-09-18T15:19:37.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-18T15:19:37.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:214: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-18T15:19:37.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-18T15:19:37.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-09-18T15:19:37.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:184: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-18T15:19:37.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local poolname=ecpool 2024-09-18T15:19:37.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local first=3 2024-09-18T15:19:37.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local second=5 2024-09-18T15:19:37.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:192: corrupt_and_repair_two: pids= 2024-09-18T15:19:37.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:19:37.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:19:37.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:19:37.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 268380"' 2024-09-18T15:19:37.230 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 268380' 2024-09-18T15:19:37.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-18T15:19:37.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:19:37.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:19:37.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 268381"' 2024-09-18T15:19:37.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:19:37.230 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 268381' 2024-09-18T15:19:37.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: wait_background pids 2024-09-18T15:19:37.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: wait_background: pids=' 268380 268381' 2024-09-18T15:19:37.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/268383: /' 2024-09-18T15:19:37.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: wait_background: return_code=0 2024-09-18T15:19:37.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:19:37.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 268380 2024-09-18T15:19:37.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-18T15:19:37.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/268385: /' 2024-09-18T15:19:39.761 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:19:39.761 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:19:39.761 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-09-18T15:19:39.761 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:19:39.761 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-18T15:19:39.761 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:19:39.761 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:19:39.761 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-09-18T15:19:39.762 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:19:39.762 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-18T15:19:39.762 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:19:39.762 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:19:39.762 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:19:39.762 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:19:39.762 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:19:39.762 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:19:39.762 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-18T15:19:39.762 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:19:39.762 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:19:39.762 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-09-18T15:19:39.762 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:19:39.763 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-18T15:19:39.763 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-18T15:19:39.763 INFO:tasks.workunit.client.0.smithi046.stderr:268385: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-18T15:19:39.763 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-18T15:19:39.763 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:19:39.763 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:19:39.763 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-09-18T15:19:39.763 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:19:39.763 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-18T15:19:39.763 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:19:39.763 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:19:39.763 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: 268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:19:39.763 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:19:39.763 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-18T15:19:39.763 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:19:39.764 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:19:39.764 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:19:39.764 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:19:39.764 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-18T15:19:39.764 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:19:39.764 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-18T15:19:39.764 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:19:39.764 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:19:39.764 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:19:39.764 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:19:39.764 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:19:39.764 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:19:39.764 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:19:39.764 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:19:39.765 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:19:39.765 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-18T15:19:39.765 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:19:39.765 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:19:39.765 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-18T15:19:39.765 INFO:tasks.workunit.client.0.smithi046.stderr:268383: remove 0#2:eb822e21:::SOMETHING:head# 2024-09-18T15:19:39.765 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-18T15:19:39.765 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:19:39.765 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:19:39.765 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T15:19:39.765 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:19:39.765 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:19:39.765 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:19:39.765 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:19:39.766 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:19:39.766 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:19:39.766 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-18T15:19:39.766 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-18T15:19:39.766 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:19:39.766 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:19:39.766 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:19:39.767 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:19:39.767 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:19:39.767 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:19:39.767 INFO:tasks.workunit.client.0.smithi046.stderr:268385: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:19:39.767 INFO:tasks.workunit.client.0.smithi046.stderr:268385: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:19:39.767 INFO:tasks.workunit.client.0.smithi046.stderr:268385: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:19:39.767 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:19:39.767 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:19:39.767 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:19:39.767 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:19:39.767 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:19:39.767 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:19:39.767 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:19:39.768 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:19:39.768 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:19:39.768 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:19:39.768 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:19:39.768 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-18T15:19:39.768 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-09-18T15:19:39.768 INFO:tasks.workunit.client.0.smithi046.stderr:268385: start osd.5 2024-09-18T15:19:39.768 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:19:39.768 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:19:39.769 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-18T15:19:39.769 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-18T15:19:39.769 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:19:39.769 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:19:39.769 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:19:39.769 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:19:39.769 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:19:39.769 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:19:39.769 INFO:tasks.workunit.client.0.smithi046.stderr:268383: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:19:39.769 INFO:tasks.workunit.client.0.smithi046.stderr:268383: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:19:39.769 INFO:tasks.workunit.client.0.smithi046.stderr:268383: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:19:39.769 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:19:39.770 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:19:39.770 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:19:39.770 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:19:39.770 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:19:39.770 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:19:39.770 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:19:39.770 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:19:39.770 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:19:39.770 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:19:39.770 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:19:39.770 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-18T15:19:39.770 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T15:19:39.770 INFO:tasks.workunit.client.0.smithi046.stderr:268383: start osd.3 2024-09-18T15:19:41.869 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:19:41.870 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-18T15:19:41.870 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T15:19:41.870 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:19:41.870 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:19:41.870 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:19:41.870 INFO:tasks.workunit.client.0.smithi046.stderr:268383: 2024-09-18T15:19:39.787+0000 7f58c20da540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:41.870 INFO:tasks.workunit.client.0.smithi046.stderr:268383: 2024-09-18T15:19:39.791+0000 7f58c20da540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:41.870 INFO:tasks.workunit.client.0.smithi046.stderr:268383: 2024-09-18T15:19:39.795+0000 7f58c20da540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:41.870 INFO:tasks.workunit.client.0.smithi046.stderr:268383: 2024-09-18T15:19:39.803+0000 7f58c20da540 -1 Falling back to public interface 2024-09-18T15:19:41.870 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T15:19:41.870 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:19:41.871 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:19:41.871 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:19:41.871 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:19:41.871 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:41.871 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:19:41.871 INFO:tasks.workunit.client.0.smithi046.stderr:268383: 0 2024-09-18T15:19:41.871 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:41.871 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:19:41.871 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:41.871 INFO:tasks.workunit.client.0.smithi046.stderr:268383: 2024-09-18T15:19:40.855+0000 7f58c20da540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:19:41.871 INFO:tasks.workunit.client.0.smithi046.stderr:268383: 2024-09-18T15:19:40.871+0000 7f58c20da540 -1 journal do_read_entry(1212416): bad header magic 2024-09-18T15:19:41.872 INFO:tasks.workunit.client.0.smithi046.stderr:268383: 2024-09-18T15:19:40.871+0000 7f58c20da540 -1 journal do_read_entry(1212416): bad header magic 2024-09-18T15:19:41.872 INFO:tasks.workunit.client.0.smithi046.stderr:268383: 2024-09-18T15:19:40.883+0000 7f58c20da540 -1 osd.3 104 log_to_monitors true 2024-09-18T15:19:41.872 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:41.872 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:41.872 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:19:41.872 INFO:tasks.workunit.client.0.smithi046.stderr:268383: 1 2024-09-18T15:19:41.872 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:41.872 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:19:41.872 INFO:tasks.workunit.client.0.smithi046.stderr:268383: osd.3 up in weight 1 up_from 108 up_thru 108 down_at 105 last_clean_interval [71,104) [v2:127.0.0.1:6826/268520,v1:127.0.0.1:6827/268520] [v2:127.0.0.1:6828/268520,v1:127.0.0.1:6829/268520] exists,up 0fcca98d-f3b0-4b6a-8ec6-ccc43b7d6de2 2024-09-18T15:19:41.872 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:19:41.872 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:19:41.872 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:19:41.872 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:19:41.873 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:19:41.873 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:19:41.873 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_ops 2024-09-18T15:19:41.874 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-18T15:19:41.874 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-09-18T15:19:41.874 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:19:41.874 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:19:41.874 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:19:41.874 INFO:tasks.workunit.client.0.smithi046.stderr:268385: 2024-09-18T15:19:39.807+0000 7fb68f94c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:41.874 INFO:tasks.workunit.client.0.smithi046.stderr:268385: 2024-09-18T15:19:39.811+0000 7fb68f94c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:41.874 INFO:tasks.workunit.client.0.smithi046.stderr:268385: 2024-09-18T15:19:39.815+0000 7fb68f94c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:41.874 INFO:tasks.workunit.client.0.smithi046.stderr:268385: 2024-09-18T15:19:39.815+0000 7fb68f94c540 -1 Falling back to public interface 2024-09-18T15:19:41.874 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-09-18T15:19:41.874 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:19:41.875 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T15:19:41.875 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:19:41.875 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:19:41.875 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:41.875 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:19:41.875 INFO:tasks.workunit.client.0.smithi046.stderr:268385: 0 2024-09-18T15:19:41.875 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:41.875 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:19:41.875 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:41.875 INFO:tasks.workunit.client.0.smithi046.stderr:268385: 2024-09-18T15:19:40.775+0000 7fb68f94c540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:19:41.875 INFO:tasks.workunit.client.0.smithi046.stderr:268385: 2024-09-18T15:19:40.779+0000 7fb68f94c540 -1 journal do_read_entry(1208320): bad header magic 2024-09-18T15:19:41.875 INFO:tasks.workunit.client.0.smithi046.stderr:268385: 2024-09-18T15:19:40.779+0000 7fb68f94c540 -1 journal do_read_entry(1208320): bad header magic 2024-09-18T15:19:41.875 INFO:tasks.workunit.client.0.smithi046.stderr:268385: 2024-09-18T15:19:40.795+0000 7fb68f94c540 -1 osd.5 104 log_to_monitors true 2024-09-18T15:19:41.876 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:41.876 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:41.876 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:19:41.876 INFO:tasks.workunit.client.0.smithi046.stderr:268385: 1 2024-09-18T15:19:41.876 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:41.876 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:19:41.876 INFO:tasks.workunit.client.0.smithi046.stderr:268385: osd.5 up in weight 1 up_from 108 up_thru 108 down_at 105 last_clean_interval [102,104) [v2:127.0.0.1:6842/268539,v1:127.0.0.1:6843/268539] [v2:127.0.0.1:6844/268539,v1:127.0.0.1:6845/268539] exists,up 581d8192-9e17-497f-95a0-795d5338b5fe 2024-09-18T15:19:41.876 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:19:41.876 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:19:41.876 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:19:41.876 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:19:41.876 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:19:41.876 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:19:43.312 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: waitfor_clean: local cur_active_clean 2024-09-18T15:19:43.312 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:19:43.312 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:19:43.313 INFO:tasks.workunit.client.0.smithi046.stderr:268383: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:19:43.313 INFO:tasks.workunit.client.0.smithi046.stderr:268383: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:19:43.313 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:19:43.313 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:19:43.313 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:19:43.313 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:19:43.313 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:19:43.313 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:19:43.313 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:19:43.313 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:19:43.313 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:19:43.313 INFO:tasks.workunit.client.0.smithi046.stderr:268383: 1 2024-09-18T15:19:43.314 INFO:tasks.workunit.client.0.smithi046.stderr:268383: 2 2024-09-18T15:19:43.314 INFO:tasks.workunit.client.0.smithi046.stderr:268383: 3 2024-09-18T15:19:43.314 INFO:tasks.workunit.client.0.smithi046.stderr:268383: 4 2024-09-18T15:19:43.314 INFO:tasks.workunit.client.0.smithi046.stderr:268383: 5 2024-09-18T15:19:43.314 INFO:tasks.workunit.client.0.smithi046.stderr:268383: 6 2024-09-18T15:19:43.314 INFO:tasks.workunit.client.0.smithi046.stderr:268383: 7 2024-09-18T15:19:43.314 INFO:tasks.workunit.client.0.smithi046.stderr:268383: 8 2024-09-18T15:19:43.314 INFO:tasks.workunit.client.0.smithi046.stderr:268383: 9' 2024-09-18T15:19:43.314 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:19:43.314 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:43.314 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:19:43.314 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836529 2024-09-18T15:19:43.314 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836529 2024-09-18T15:19:43.315 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836529' 2024-09-18T15:19:43.315 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:43.315 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:19:43.315 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673008 2024-09-18T15:19:43.315 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673008 2024-09-18T15:19:43.315 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836529 1-42949673008' 2024-09-18T15:19:43.315 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:43.315 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:19:43.315 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509487 2024-09-18T15:19:43.315 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509487 2024-09-18T15:19:43.315 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836529 1-42949673008 2-64424509487' 2024-09-18T15:19:43.315 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:43.315 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:19:43.315 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=463856467970 2024-09-18T15:19:43.316 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 463856467970 2024-09-18T15:19:43.323 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubun_for_clean: local cur_active_clean 2024-09-18T15:19:43.323 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:19:43.323 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:19:43.324 INFO:tasks.workunit.client.0.smithi046.stderr:268385: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:19:43.324 INFO:tasks.workunit.client.0.smithi046.stderr:268385: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:19:43.324 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:19:43.324 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:19:43.324 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:19:43.324 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:19:43.324 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:19:43.324 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:19:43.324 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:19:43.324 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:19:43.324 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:19:43.324 INFO:tasks.workunit.client.0.smithi046.stderr:268385: 1 2024-09-18T15:19:43.324 INFO:tasks.workunit.client.0.smithi046.stderr:268385: 2 2024-09-18T15:19:43.325 INFO:tasks.workunit.client.0.smithi046.stderr:268385: 3 2024-09-18T15:19:43.325 INFO:tasks.workunit.client.0.smithi046.stderr:268385: 4 2024-09-18T15:19:43.325 INFO:tasks.workunit.client.0.smithi046.stderr:268385: 5 2024-09-18T15:19:43.325 INFO:tasks.workunit.client.0.smithi046.stderr:268385: 6 2024-09-18T15:19:43.325 INFO:tasks.workunit.client.0.smithi046.stderr:268385: 7 2024-09-18T15:19:43.325 INFO:tasks.workunit.client.0.smithi046.stderr:268385: 8 2024-09-18T15:19:43.325 INFO:tasks.workunit.client.0.smithi046.stderr:268385: 9' 2024-09-18T15:19:43.325 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:19:43.325 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:43.325 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:19:43.325 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836528 2024-09-18T15:19:43.325 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836528 2024-09-18T15:19:43.325 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836528' 2024-09-18T15:19:43.325 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:43.326 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:19:43.326 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673007 2024-09-18T15:19:43.326 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673007 2024-09-18T15:19:43.326 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836528 1-42949673007' 2024-09-18T15:19:43.326 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:43.326 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:19:43.326 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509488 2024-09-18T15:19:43.326 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509488 2024-09-18T15:19:43.326 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836528 1-42949673007 2-64424509488' 2024-09-18T15:19:43.326 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:43.326 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:19:43.326 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=463856467971 2024-09-18T15:19:43.326 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 463856467971 2024-09-18T15:19:44.087 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubutu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836529 1-42949673008 2-64424509487 3-463856467970' 2024-09-18T15:19:44.087 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:44.087 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:19:44.088 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182445 2024-09-18T15:19:44.088 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182445 2024-09-18T15:19:44.088 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836529 1-42949673008 2-64424509487 3-463856467970 4-107374182445' 2024-09-18T15:19:44.088 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:44.088 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:19:44.088 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=463856467970 2024-09-18T15:19:44.088 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 463856467970 2024-09-18T15:19:44.088 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836529 1-42949673008 2-64424509487 3-463856467970 4-107374182445 5-463856467970' 2024-09-18T15:19:44.088 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:44.088 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:19:44.088 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855402 2024-09-18T15:19:44.088 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855402 2024-09-18T15:19:44.089 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836529 1-42949673008 2-64424509487 3-463856467970 4-107374182445 5-463856467970 6-150323855402' 2024-09-18T15:19:44.089 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:44.089 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:19:44.089 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691882 2024-09-18T15:19:44.089 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691882 2024-09-18T15:19:44.089 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836529 1-42949673008 2-64424509487 3-463856467970 4-107374182445 5-463856467970 6-150323855402 7-171798691882' 2024-09-18T15:19:44.089 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:44.089 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:19:44.089 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528361 2024-09-18T15:19:44.089 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528361 2024-09-18T15:19:44.089 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836529 1-42949673008 2-64424509487 3-463856467970 4-107374182445 5-463856467970 6-150323855402 7-171798691882 8-193273528361' 2024-09-18T15:19:44.090 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:44.090 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:19:44.090 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=438086664198 2024-09-18T15:19:44.090 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 438086664198 2024-09-18T15:19:44.090 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.ntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836528 1-42949673007 2-64424509488 3-463856467971' 2024-09-18T15:19:44.090 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:44.090 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:19:44.090 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182446 2024-09-18T15:19:44.090 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182446 2024-09-18T15:19:44.091 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836528 1-42949673007 2-64424509488 3-463856467971 4-107374182446' 2024-09-18T15:19:44.091 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:44.091 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:19:44.091 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=463856467971 2024-09-18T15:19:44.091 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 463856467971 2024-09-18T15:19:44.091 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836528 1-42949673007 2-64424509488 3-463856467971 4-107374182446 5-463856467971' 2024-09-18T15:19:44.091 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:44.091 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:19:44.091 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855403 2024-09-18T15:19:44.091 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855403 2024-09-18T15:19:44.091 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836528 1-42949673007 2-64424509488 3-463856467971 4-107374182446 5-463856467971 6-150323855403' 2024-09-18T15:19:44.091 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:44.091 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:19:44.091 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691883 2024-09-18T15:19:44.092 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691883 2024-09-18T15:19:44.092 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836528 1-42949673007 2-64424509488 3-463856467971 4-107374182446 5-463856467971 6-150323855403 7-171798691883' 2024-09-18T15:19:44.092 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:44.092 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:19:44.092 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528360 2024-09-18T15:19:44.092 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528360 2024-09-18T15:19:44.092 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836528 1-42949673007 2-64424509488 3-463856467971 4-107374182446 5-463856467971 6-150323855403 7-171798691883 8-193273528360' 2024-09-18T15:19:44.092 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:44.092 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:19:44.092 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=438086664199 2024-09-18T15:19:44.092 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 438086664199 2024-09-18T15:19:45.193 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836529 1-42949673008 2-64424509487 3-463856467970 4-107374182445 5-463856467970 6-150323855402 7-171798691882 8-193273528361 9-438086664198' 2024-09-18T15:19:45.194 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:45.194 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836529 2024-09-18T15:19:45.194 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:45.194 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:19:45.194 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:45.194 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836529 2024-09-18T15:19:45.194 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836529 2024-09-18T15:19:45.194 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2024-09-18T15:19:45.194 INFO:tasks.workunit.client.0.smithi046.stderr:268383: waiting osd.0 seq 21474836529 2024-09-18T15:19:45.194 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:19:45.194 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836529 -lt 21474836529 2024-09-18T15:19:45.194 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:45.194 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673008 2024-09-18T15:19:45.195 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:45.195 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:19:45.195 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673008 2024-09-18T15:19:45.195 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:45.195 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673008 2024-09-18T15:19:45.195 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673008' 2024-09-18T15:19:45.195 INFO:tasks.workunit.client.0.smithi046.stderr:268383: waiting osd.1 seq 42949673008 2024-09-18T15:19:45.195 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:19:45.195 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673009 -lt 42949673008 2024-09-18T15:19:45.195 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:45.195 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:45.195 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509487 2024-09-18T15:19:45.195 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:19:45.195 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:45.195 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509487 2024-09-18T15:19:45.196 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509487 2024-09-18T15:19:45.196 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509487' 2024-09-18T15:19:45.196 INFO:tasks.workunit.client.0.smithi046.stderr:268383: waiting osd.2 seq 64424509487 2024-09-18T15:19:45.196 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:19:45.196 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509489 -lt 64424509487 2024-09-18T15:19:45.196 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:45.202 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-4638.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836528 1-42949673007 2-64424509488 3-463856467971 4-107374182446 5-463856467971 6-150323855403 7-171798691883 8-193273528360 9-438086664199' 2024-09-18T15:19:45.202 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:45.202 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836528 2024-09-18T15:19:45.202 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:45.203 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:19:45.203 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836528 2024-09-18T15:19:45.203 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:45.203 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836528 2024-09-18T15:19:45.203 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836528' 2024-09-18T15:19:45.203 INFO:tasks.workunit.client.0.smithi046.stderr:268385: waiting osd.0 seq 21474836528 2024-09-18T15:19:45.203 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:19:45.203 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836529 -lt 21474836528 2024-09-18T15:19:45.203 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:45.203 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673007 2024-09-18T15:19:45.203 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:45.203 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:19:45.203 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673007 2024-09-18T15:19:45.203 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:45.204 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673007 2024-09-18T15:19:45.204 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673007' 2024-09-18T15:19:45.204 INFO:tasks.workunit.client.0.smithi046.stderr:268385: waiting osd.1 seq 42949673007 2024-09-18T15:19:45.204 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:19:45.204 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673009 -lt 42949673007 2024-09-18T15:19:45.204 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:45.204 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509488 2024-09-18T15:19:45.204 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:45.204 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:19:45.204 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:45.204 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509488 2024-09-18T15:19:45.204 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509488 2024-09-18T15:19:45.204 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509488' 2024-09-18T15:19:45.205 INFO:tasks.workunit.client.0.smithi046.stderr:268385: waiting osd.2 seq 64424509488 2024-09-18T15:19:45.205 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:19:45.205 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509489 -lt 64424509488 2024-09-18T15:19:45.205 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:46.356 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-463856467971 2024-09-18T15:19:46.356 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:46.356 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:19:46.356 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:46.356 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-463856467971 2024-09-18T15:19:46.356 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=463856467971 2024-09-18T15:19:46.356 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 463856467971' 2024-09-18T15:19:46.356 INFO:tasks.workunit.client.0.smithi046.stderr:268385: waiting osd.3 seq 463856467971 2024-09-18T15:19:46.356 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:19:46.356 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 463856467971 -lt 463856467971 2024-09-18T15:19:46.357 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:46.357 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182446 2024-09-18T15:19:46.357 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:46.357 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:19:46.357 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182446 2024-09-18T15:19:46.357 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:46.357 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182446 2024-09-18T15:19:46.357 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182446' 2024-09-18T15:19:46.357 INFO:tasks.workunit.client.0.smithi046.stderr:268385: waiting osd.4 seq 107374182446 2024-09-18T15:19:46.357 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:19:46.357 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182446 -lt 107374182446 2024-09-18T15:19:46.357 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:46.357 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-463856467971 2024-09-18T15:19:46.358 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:46.358 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:19:46.358 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-463856467971 2024-09-18T15:19:46.358 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:46.358 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=463856467971 2024-09-18T15:19:46.358 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 463856467971' 2024-09-18T15:19:46.358 INFO:tasks.workunit.client.0.smithi046.stderr:268385: waiting osd.5 seq 463856467971 2024-09-18T15:19:46.358 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:19:46.358 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 463856467971 -lt 463856467971 2024-09-18T15:19:46.358 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:46.358 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855403 2024-09-18T15:19:46.358 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:46.358 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:19:46.358 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855403 2024-09-18T15:19:46.359 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-h56467970 2024-09-18T15:19:46.359 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:46.359 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:19:46.359 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-463856467970 2024-09-18T15:19:46.359 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:46.359 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=463856467970 2024-09-18T15:19:46.359 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 463856467970' 2024-09-18T15:19:46.359 INFO:tasks.workunit.client.0.smithi046.stderr:268383: waiting osd.3 seq 463856467970 2024-09-18T15:19:46.359 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:19:46.359 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 463856467971 -lt 463856467970 2024-09-18T15:19:46.360 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:46.360 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182445 2024-09-18T15:19:46.360 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:46.360 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:19:46.360 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182445 2024-09-18T15:19:46.360 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:46.360 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182445 2024-09-18T15:19:46.360 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182445' 2024-09-18T15:19:46.360 INFO:tasks.workunit.client.0.smithi046.stderr:268383: waiting osd.4 seq 107374182445 2024-09-18T15:19:46.361 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:19:46.361 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182446 -lt 107374182445 2024-09-18T15:19:46.361 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:46.361 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-463856467970 2024-09-18T15:19:46.361 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:46.361 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:19:46.361 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-463856467970 2024-09-18T15:19:46.361 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:46.361 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=463856467970 2024-09-18T15:19:46.361 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 463856467970' 2024-09-18T15:19:46.361 INFO:tasks.workunit.client.0.smithi046.stderr:268383: waiting osd.5 seq 463856467970 2024-09-18T15:19:46.361 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:19:46.362 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 463856467971 -lt 463856467970 2024-09-18T15:19:46.362 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:46.362 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855402 2024-09-18T15:19:46.362 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:46.362 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:19:46.362 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855402 2024-09-18T15:19:47.395 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-heelpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:47.395 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855403 2024-09-18T15:19:47.395 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855403' 2024-09-18T15:19:47.395 INFO:tasks.workunit.client.0.smithi046.stderr:268385: waiting osd.6 seq 150323855403 2024-09-18T15:19:47.395 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:19:47.395 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855404 -lt 150323855403 2024-09-18T15:19:47.395 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:47.396 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:47.396 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691883 2024-09-18T15:19:47.396 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:19:47.396 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691883 2024-09-18T15:19:47.396 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:47.396 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691883 2024-09-18T15:19:47.396 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691883' 2024-09-18T15:19:47.396 INFO:tasks.workunit.client.0.smithi046.stderr:268385: waiting osd.7 seq 171798691883 2024-09-18T15:19:47.396 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:19:47.396 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691883 -lt 171798691883 2024-09-18T15:19:47.396 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:47.397 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528360 2024-09-18T15:19:47.397 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:47.397 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:19:47.397 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528360 2024-09-18T15:19:47.397 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:47.397 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528360 2024-09-18T15:19:47.397 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528360' 2024-09-18T15:19:47.397 INFO:tasks.workunit.client.0.smithi046.stderr:268385: waiting osd.8 seq 193273528360 2024-09-18T15:19:47.397 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:19:47.397 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528361 -lt 193273528360 2024-09-18T15:19:47.397 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:47.397 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-438086664199 2024-09-18T15:19:47.397 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:47.398 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:19:47.398 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-438086664199 2024-09-18T15:19:47.398 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:47.398 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=438086664199 2024-09-18T15:19:47.398 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 438086664199' 2024-09-18T15:19:47.398 INFO:tasks.workunit.client.0.smithi046.stderr:268385: waiting osd.9 seq 438086664199 2024-09-18T15:19:47.401 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.shlpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:47.401 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855402 2024-09-18T15:19:47.401 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855402' 2024-09-18T15:19:47.401 INFO:tasks.workunit.client.0.smithi046.stderr:268383: waiting osd.6 seq 150323855402 2024-09-18T15:19:47.401 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:19:47.401 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855404 -lt 150323855402 2024-09-18T15:19:47.402 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:47.402 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691882 2024-09-18T15:19:47.402 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:47.402 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:19:47.402 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691882 2024-09-18T15:19:47.402 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:47.402 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691882 2024-09-18T15:19:47.402 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691882' 2024-09-18T15:19:47.402 INFO:tasks.workunit.client.0.smithi046.stderr:268383: waiting osd.7 seq 171798691882 2024-09-18T15:19:47.402 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:19:47.402 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691883 -lt 171798691882 2024-09-18T15:19:47.402 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:47.402 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528361 2024-09-18T15:19:47.403 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:47.403 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:19:47.403 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528361 2024-09-18T15:19:47.403 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:47.403 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528361 2024-09-18T15:19:47.403 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528361' 2024-09-18T15:19:47.403 INFO:tasks.workunit.client.0.smithi046.stderr:268383: waiting osd.8 seq 193273528361 2024-09-18T15:19:47.403 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:19:47.403 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528361 -lt 193273528361 2024-09-18T15:19:47.403 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:19:47.403 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-438086664198 2024-09-18T15:19:47.403 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:19:47.403 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:19:47.404 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-438086664198 2024-09-18T15:19:47.404 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:19:47.404 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=438086664198 2024-09-18T15:19:47.404 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 438086664198' 2024-09-18T15:19:47.404 INFO:tasks.workunit.client.0.smithi046.stderr:268383: waiting osd.9 seq 438086664198 2024-09-18T15:19:48.836 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:19:48.836 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 438086664200 -lt 438086664198 2024-09-18T15:19:48.836 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:19:48.836 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:19:48.836 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:19:48.836 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:19:48.836 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:19:48.836 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:19:48.836 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:19:48.836 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:19:48.837 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:19:48.837 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:19:48.837 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:19:48.837 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:19:48.837 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:19:48.837 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:19:48.837 INFO:tasks.workunit.client.0.smithi046.stderr:268383: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:19:48.837 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:19:48.837 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:19:48.837 INFO:tasks.workunit.client.0.smithi046.stderr:268383: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:19:48.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:19:48.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:19:48.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 268381 2024-09-18T15:19:48.847 INFO:tasks.workunit.client.0.smithi046.stderr::2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:19:48.848 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 438086664200 -lt 438086664199 2024-09-18T15:19:48.848 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:19:48.848 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:19:48.848 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:19:48.848 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:19:48.848 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:19:48.848 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:19:48.848 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:19:48.848 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:19:48.848 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:19:48.849 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:19:48.849 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:19:48.849 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:19:48.849 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:19:48.849 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:19:48.850 INFO:tasks.workunit.client.0.smithi046.stderr:268385: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:19:48.850 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:19:48.850 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:19:48.850 INFO:tasks.workunit.client.0.smithi046.stderr:268385: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:19:48.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:19:48.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: eval 'pids='\'''\''' 2024-09-18T15:19:48.851 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: pids= 2024-09-18T15:19:48.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: wait_background: return 0 2024-09-18T15:19:48.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: return_code=0 2024-09-18T15:19:48.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-18T15:19:48.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-18T15:19:48.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-09-18T15:19:48.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T15:19:48.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:19:48.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:19:49.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: local pg=2.0 2024-09-18T15:19:49.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: repair 2.0 2024-09-18T15:19:49.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-09-18T15:19:49.198 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-09-18T15:19:49.198 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:19:49.198 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:19:49.199 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:19:49.199 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:19:49.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T15:19:18.125540+0000 2024-09-18T15:19:49.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-09-18T15:19:49.830 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-18T15:19:49.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-09-18T15:19:18.125540+0000 2024-09-18T15:19:49.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=2.0 2024-09-18T15:19:49.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:19:18.125540+0000 2024-09-18T15:19:49.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:19:49.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:19:49.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:19:49.843 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:19:49.843 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:19:49.844 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:19:49.844 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:19:49.844 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:19:50.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:19:18.125540+0000 '>' 2024-09-18T15:19:18.125540+0000 2024-09-18T15:19:50.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:19:51.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:19:51.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:19:51.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:19:51.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:19:51.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:19:51.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:19:51.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:19:51.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:19:18.125540+0000 '>' 2024-09-18T15:19:18.125540+0000 2024-09-18T15:19:51.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:19:52.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:19:52.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:19:52.491 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:19:52.491 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:19:52.492 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:19:52.492 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:19:52.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:19:52.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:19:50.055316+0000 '>' 2024-09-18T15:19:18.125540+0000 2024-09-18T15:19:52.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:19:52.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:207: corrupt_and_repair_two: pids= 2024-09-18T15:19:52.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:19:52.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:19:52.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:19:52.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 270426"' 2024-09-18T15:19:52.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 270426' 2024-09-18T15:19:52.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-18T15:19:52.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:19:52.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:19:52.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:19:52.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/270428: /' 2024-09-18T15:19:52.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 270429"' 2024-09-18T15:19:52.814 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 270429' 2024-09-18T15:19:52.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: wait_background pids 2024-09-18T15:19:52.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: wait_background: pids=' 270426 270429' 2024-09-18T15:19:52.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: wait_background: return_code=0 2024-09-18T15:19:52.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:19:52.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 270426 2024-09-18T15:19:52.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/270432: /' 2024-09-18T15:19:52.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-18T15:19:54.458 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:19:54.458 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:19:54.458 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-18T15:19:54.458 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:19:54.458 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:19:54.458 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:19:54.458 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:19:54.458 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-18T15:19:54.458 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:19:54.459 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-18T15:19:54.459 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:19:54.459 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:19:54.459 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:19:54.459 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:19:54.459 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:19:54.459 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:19:54.459 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:19:54.459 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:19:54.459 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:19:54.459 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-18T15:19:54.459 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:19:54.459 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:19:54.459 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-18T15:19:54.460 INFO:tasks.workunit.client.0.smithi046.stderr:270428: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-18T15:19:54.460 INFO:tasks.workunit.client.0.smithi046.stderr:270428: _ 2024-09-18T15:19:54.460 INFO:tasks.workunit.client.0.smithi046.stderr:270428: hinfo_key 2024-09-18T15:19:54.460 INFO:tasks.workunit.client.0.smithi046.stderr:270428: snapset 2024-09-18T15:19:54.460 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-18T15:19:54.460 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:19:54.460 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:19:54.460 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T15:19:54.460 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:19:54.460 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:19:54.460 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:19:54.465 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:19:54.465 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:19:54.467 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:19:54.468 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-18T15:19:54.468 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-18T15:19:54.468 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:19:54.468 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:19:54.468 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:19:54.468 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:19:54.468 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:19:54.468 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:19:54.468 INFO:tasks.workunit.client.0.smithi046.stderr:270428: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:19:54.468 INFO:tasks.workunit.client.0.smithi046.stderr:270428: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:19:54.468 INFO:tasks.workunit.client.0.smithi046.stderr:270428: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:19:54.468 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:19:54.469 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:19:54.469 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:19:54.469 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:19:54.469 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:19:54.469 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:19:54.469 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:19:54.469 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:19:54.469 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:19:54.469 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:19:54.469 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:19:54.469 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-18T15:19:54.469 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T15:19:54.469 INFO:tasks.workunit.client.0.smithi046.stderr:270428: start osd.3 2024-09-18T15:19:54.601 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namesp270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:19:54.601 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:19:54.601 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-09-18T15:19:54.601 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:19:54.601 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-18T15:19:54.601 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:19:54.601 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:19:54.602 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-09-18T15:19:54.602 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:19:54.602 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-18T15:19:54.602 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:19:54.602 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:19:54.602 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:19:54.602 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:19:54.602 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:19:54.602 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:19:54.602 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-18T15:19:54.602 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:19:54.602 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:19:54.602 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-09-18T15:19:54.603 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:19:54.603 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-18T15:19:54.603 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-09-18T15:19:54.603 INFO:tasks.workunit.client.0.smithi046.stderr:270432: _ 2024-09-18T15:19:54.603 INFO:tasks.workunit.client.0.smithi046.stderr:270432: hinfo_key 2024-09-18T15:19:54.603 INFO:tasks.workunit.client.0.smithi046.stderr:270432: snapset 2024-09-18T15:19:54.603 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-18T15:19:54.603 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:19:54.603 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:19:54.603 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-09-18T15:19:54.603 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:19:54.603 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-18T15:19:54.603 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:19:54.604 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:19:54.612 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:19:54.612 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:19:54.612 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-18T15:19:54.612 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-18T15:19:54.612 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:19:54.612 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:19:54.613 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:19:54.613 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:19:54.613 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:19:54.613 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:19:54.613 INFO:tasks.workunit.client.0.smithi046.stderr:270432: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:19:54.613 INFO:tasks.workunit.client.0.smithi046.stderr:270432: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:19:54.613 INFO:tasks.workunit.client.0.smithi046.stderr:270432: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:19:54.613 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:19:54.613 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:19:54.613 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:19:54.613 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:19:54.613 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:19:54.613 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:19:54.614 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:19:54.614 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:19:54.614 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:19:54.614 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:19:54.614 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:19:54.614 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-18T15:19:54.614 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-09-18T15:19:54.614 INFO:tasks.workunit.client.0.smithi046.stderr:270432: start osd.5 2024-09-18T15:19:54.614 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-18T15:19:57.236 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-lenace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:19:57.236 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-18T15:19:57.237 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T15:19:57.237 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:19:57.237 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:19:57.237 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:19:57.237 INFO:tasks.workunit.client.0.smithi046.stderr:270428: 2024-09-18T15:19:54.491+0000 7f459771b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:57.237 INFO:tasks.workunit.client.0.smithi046.stderr:270428: 2024-09-18T15:19:54.591+0000 7f459771b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:57.237 INFO:tasks.workunit.client.0.smithi046.stderr:270428: 2024-09-18T15:19:54.595+0000 7f459771b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:57.237 INFO:tasks.workunit.client.0.smithi046.stderr:270428: 2024-09-18T15:19:54.599+0000 7f459771b540 -1 Falling back to public interface 2024-09-18T15:19:57.237 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T15:19:57.237 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:19:57.237 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:19:57.237 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:19:57.238 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:19:57.238 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:57.238 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:19:57.238 INFO:tasks.workunit.client.0.smithi046.stderr:270428: 0 2024-09-18T15:19:57.238 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:57.238 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:19:57.238 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:57.238 INFO:tasks.workunit.client.0.smithi046.stderr:270428: 2024-09-18T15:19:55.847+0000 7f459771b540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:19:57.238 INFO:tasks.workunit.client.0.smithi046.stderr:270428: 2024-09-18T15:19:55.855+0000 7f459771b540 -1 journal do_read_entry(1306624): bad header magic 2024-09-18T15:19:57.238 INFO:tasks.workunit.client.0.smithi046.stderr:270428: 2024-09-18T15:19:55.855+0000 7f459771b540 -1 journal do_read_entry(1306624): bad header magic 2024-09-18T15:19:57.238 INFO:tasks.workunit.client.0.smithi046.stderr:270428: 2024-09-18T15:19:55.883+0000 7f459771b540 -1 osd.3 109 log_to_monitors true 2024-09-18T15:19:57.238 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:57.238 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:57.238 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:19:57.239 INFO:tasks.workunit.client.0.smithi046.stderr:270428: 1 2024-09-18T15:19:57.239 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:57.239 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:19:57.239 INFO:tasks.workunit.client.0.smithi046.stderr:270428: osd.3 up in weight 1 up_from 114 up_thru 108 down_at 110 last_clean_interval [108,109) [v2:127.0.0.1:6826/270576,v1:127.0.0.1:6827/270576] [v2:127.0.0.1:6828/270576,v1:127.0.0.1:6829/270576] exists,up 0fcca98d-f3b0-4b6a-8ec6-ccc43b7d6de2 2024-09-18T15:19:57.239 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:19:57.239 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:19:57.239 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:19:57.239 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:19:57.239 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:19:57.239 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: =460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:19:57.239 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-09-18T15:19:57.239 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:19:57.239 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:19:57.240 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:19:57.240 INFO:tasks.workunit.client.0.smithi046.stderr:270432: 2024-09-18T15:19:54.631+0000 7f620d3ff540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:57.240 INFO:tasks.workunit.client.0.smithi046.stderr:270432: 2024-09-18T15:19:54.635+0000 7f620d3ff540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:57.240 INFO:tasks.workunit.client.0.smithi046.stderr:270432: 2024-09-18T15:19:54.639+0000 7f620d3ff540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:57.240 INFO:tasks.workunit.client.0.smithi046.stderr:270432: 2024-09-18T15:19:54.639+0000 7f620d3ff540 -1 Falling back to public interface 2024-09-18T15:19:57.240 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-09-18T15:19:57.240 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:19:57.240 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T15:19:57.240 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:19:57.240 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:19:57.240 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:57.240 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:19:57.240 INFO:tasks.workunit.client.0.smithi046.stderr:270432: 0 2024-09-18T15:19:57.240 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:57.241 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:19:57.241 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:57.241 INFO:tasks.workunit.client.0.smithi046.stderr:270432: 2024-09-18T15:19:55.847+0000 7f620d3ff540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:19:57.241 INFO:tasks.workunit.client.0.smithi046.stderr:270432: 2024-09-18T15:19:55.855+0000 7f620d3ff540 -1 journal do_read_entry(1314816): bad header magic 2024-09-18T15:19:57.241 INFO:tasks.workunit.client.0.smithi046.stderr:270432: 2024-09-18T15:19:55.855+0000 7f620d3ff540 -1 journal do_read_entry(1314816): bad header magic 2024-09-18T15:19:57.241 INFO:tasks.workunit.client.0.smithi046.stderr:270432: 2024-09-18T15:19:55.883+0000 7f620d3ff540 -1 osd.5 109 log_to_monitors true 2024-09-18T15:19:57.241 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:57.241 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:57.241 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:19:57.241 INFO:tasks.workunit.client.0.smithi046.stderr:270432: 1 2024-09-18T15:19:57.241 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:57.241 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:19:57.241 INFO:tasks.workunit.client.0.smithi046.stderr:270432: osd.5 up in weight 1 up_from 114 up_thru 108 down_at 110 last_clean_interval [108,109) [v2:127.0.0.1:6842/270612,v1:127.0.0.1:6843/270612] [v2:127.0.0.1:6844/270612,v1:127.0.0.1:6845/270612] exists,up 581d8192-9e17-497f-95a0-795d5338b5fe 2024-09-18T15:19:57.242 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:19:57.242 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:19:57.242 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:19:57.242 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:19:57.242 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:19:57.242 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:19:58.493 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:local num_active_clean=-1 2024-09-18T15:19:58.494 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:19:58.494 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:19:58.494 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:19:58.494 INFO:tasks.workunit.client.0.smithi046.stderr:270428: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:19:58.494 INFO:tasks.workunit.client.0.smithi046.stderr:270428: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:19:58.494 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:19:58.494 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:19:58.494 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:19:58.494 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:19:58.494 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:19:58.495 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:19:58.495 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:19:58.495 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:19:58.495 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:19:58.495 INFO:tasks.workunit.client.0.smithi046.stderr:270428: 1 2024-09-18T15:19:58.495 INFO:tasks.workunit.client.0.smithi046.stderr:270428: 2 2024-09-18T15:19:58.495 INFO:tasks.workunit.client.0.smithi046.stderr:270428: 3 2024-09-18T15:19:58.495 INFO:tasks.workunit.client.0.smithi046.stderr:270428: 4 2024-09-18T15:19:58.495 INFO:tasks.workunit.client.0.smithi046.stderr:270428: 5 2024-09-18T15:19:58.495 INFO:tasks.workunit.client.0.smithi046.stderr:270428: 6 2024-09-18T15:19:58.495 INFO:tasks.workunit.client.0.smithi046.stderr:270428: 7 2024-09-18T15:19:58.495 INFO:tasks.workunit.client.0.smithi046.stderr:270428: 8 2024-09-18T15:19:58.495 INFO:tasks.workunit.client.0.smithi046.stderr:270428: 9' 2024-09-18T15:19:58.496 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:19:58.496 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:58.496 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:19:58.496 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836533 2024-09-18T15:19:58.496 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836533 2024-09-18T15:19:58.496 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836533' 2024-09-18T15:19:58.496 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:58.496 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:19:58.496 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673013 2024-09-18T15:19:58.496 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673013 2024-09-18T15:19:58.496 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836533 1-42949673013' 2024-09-18T15:19:58.496 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:58.496 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:19:58.497 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509492 2024-09-18T15:19:58.497 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509492 2024-09-18T15:19:58.497 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836533 1-42949673013 2-64424509492' 2024-09-18T15:19:58.497 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:58.497 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:19:58.497 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=489626271746 2024-09-18T15:19:58.501 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/ceph1644: wait_for_clean: local cur_active_clean 2024-09-18T15:19:58.501 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:19:58.501 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:19:58.501 INFO:tasks.workunit.client.0.smithi046.stderr:270432: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:19:58.501 INFO:tasks.workunit.client.0.smithi046.stderr:270432: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:19:58.501 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:19:58.501 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:19:58.501 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:19:58.501 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:19:58.501 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:19:58.501 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:19:58.501 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:19:58.502 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:19:58.502 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:19:58.502 INFO:tasks.workunit.client.0.smithi046.stderr:270432: 1 2024-09-18T15:19:58.502 INFO:tasks.workunit.client.0.smithi046.stderr:270432: 2 2024-09-18T15:19:58.502 INFO:tasks.workunit.client.0.smithi046.stderr:270432: 3 2024-09-18T15:19:58.502 INFO:tasks.workunit.client.0.smithi046.stderr:270432: 4 2024-09-18T15:19:58.502 INFO:tasks.workunit.client.0.smithi046.stderr:270432: 5 2024-09-18T15:19:58.502 INFO:tasks.workunit.client.0.smithi046.stderr:270432: 6 2024-09-18T15:19:58.502 INFO:tasks.workunit.client.0.smithi046.stderr:270432: 7 2024-09-18T15:19:58.502 INFO:tasks.workunit.client.0.smithi046.stderr:270432: 8 2024-09-18T15:19:58.502 INFO:tasks.workunit.client.0.smithi046.stderr:270432: 9' 2024-09-18T15:19:58.502 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:19:58.502 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:58.503 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:19:58.503 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836534 2024-09-18T15:19:58.503 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836534 2024-09-18T15:19:58.503 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836534' 2024-09-18T15:19:58.503 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:58.503 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:19:58.503 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673012 2024-09-18T15:19:58.503 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673012 2024-09-18T15:19:58.503 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836534 1-42949673012' 2024-09-18T15:19:58.503 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:58.503 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:19:58.503 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509493 2024-09-18T15:19:58.503 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509493 2024-09-18T15:19:58.503 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-64424509493' 2024-09-18T15:19:58.504 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:58.504 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:19:58.504 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=489626271747 2024-09-18T15:19:58.504 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 489626271747 2024-09-18T15:19:59.283 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-64424509493 3-489626271747' 2024-09-18T15:19:59.283 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:59.283 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:19:59.284 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182451 2024-09-18T15:19:59.284 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182451 2024-09-18T15:19:59.284 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-64424509493 3-489626271747 4-107374182451' 2024-09-18T15:19:59.284 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:59.284 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:19:59.284 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=489626271747 2024-09-18T15:19:59.284 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 489626271747 2024-09-18T15:19:59.284 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-64424509493 3-489626271747 4-107374182451 5-489626271747' 2024-09-18T15:19:59.284 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:59.284 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:19:59.284 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855408 2024-09-18T15:19:59.284 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855408 2024-09-18T15:19:59.284 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-64424509493 3-489626271747 4-107374182451 5-489626271747 6-150323855408' 2024-09-18T15:19:59.285 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:59.285 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:19:59.285 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691887 2024-09-18T15:19:59.285 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691887 2024-09-18T15:19:59.285 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-64424509493 3-489626271747 4-107374182451 5-489626271747 6-150323855408 7-171798691887' 2024-09-18T15:19:59.285 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:59.285 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:19:59.285 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528365 2024-09-18T15:19:59.285 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528365 2024-09-18T15:19:59.285 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-64424509493 3-489626271747 4-107374182451 5-489626271747 6-150323855408 7-171798691887 8-193273528365' 2024-09-18T15:19:59.285 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:59.285 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:19:59.285 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=438086664204 2024-09-18T15:19:59.286 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 438086664204 2024-09-18T15:19:59.286 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/cltest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 489626271746 2024-09-18T15:19:59.286 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836533 1-42949673013 2-64424509492 3-489626271746' 2024-09-18T15:19:59.286 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:59.286 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:19:59.286 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182450 2024-09-18T15:19:59.286 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182450 2024-09-18T15:19:59.286 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836533 1-42949673013 2-64424509492 3-489626271746 4-107374182450' 2024-09-18T15:19:59.286 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:59.286 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:19:59.287 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=489626271746 2024-09-18T15:19:59.287 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 489626271746 2024-09-18T15:19:59.287 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836533 1-42949673013 2-64424509492 3-489626271746 4-107374182450 5-489626271746' 2024-09-18T15:19:59.287 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:59.287 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:19:59.287 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855407 2024-09-18T15:19:59.287 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855407 2024-09-18T15:19:59.287 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836533 1-42949673013 2-64424509492 3-489626271746 4-107374182450 5-489626271746 6-150323855407' 2024-09-18T15:19:59.287 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:59.287 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:19:59.287 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691888 2024-09-18T15:19:59.287 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691888 2024-09-18T15:19:59.287 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836533 1-42949673013 2-64424509492 3-489626271746 4-107374182450 5-489626271746 6-150323855407 7-171798691888' 2024-09-18T15:19:59.288 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:59.288 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:19:59.288 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528366 2024-09-18T15:19:59.288 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528366 2024-09-18T15:19:59.288 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836533 1-42949673013 2-64424509492 3-489626271746 4-107374182450 5-489626271746 6-150323855407 7-171798691888 8-193273528366' 2024-09-18T15:19:59.288 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:19:59.288 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:19:59.288 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=438086664205 2024-09-18T15:20:01.351 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 438086664205 2024-09-18T15:20:01.352 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836533 1-42949673013 2-64424509492 3-489626271746 4-107374182450 5-489626271746 6-150323855407 7-171798691888 8-193273528366 9-438086664205' 2024-09-18T15:20:01.352 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:20:01.352 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836533 2024-09-18T15:20:01.352 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:20:01.352 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:20:01.352 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836533 2024-09-18T15:20:01.352 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:20:01.352 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836533 2024-09-18T15:20:01.352 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2024-09-18T15:20:01.352 INFO:tasks.workunit.client.0.smithi046.stderr:270428: waiting osd.0 seq 21474836533 2024-09-18T15:20:01.353 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:20:01.353 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836534 -lt 21474836533 2024-09-18T15:20:01.353 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:20:01.353 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673013 2024-09-18T15:20:01.353 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:20:01.353 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:20:01.353 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673013 2024-09-18T15:20:01.353 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:20:01.353 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673013 2024-09-18T15:20:01.353 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673013' 2024-09-18T15:20:01.354 INFO:tasks.workunit.client.0.smithi046.stderr:270428: waiting osd.1 seq 42949673013 2024-09-18T15:20:01.354 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:20:01.354 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673013 -lt 42949673013 2024-09-18T15:20:01.354 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:20:01.354 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509492 2024-09-18T15:20:01.354 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:20:01.354 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:20:01.354 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509492 2024-09-18T15:20:01.354 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:20:01.354 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509492 2024-09-18T15:20:01.354 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509492' 2024-09-18T15:20:01.354 INFO:tasks.workunit.client.0.smithi046.stderr:270428: waiting osd.2 seq 64424509492 2024-09-18T15:20:01.354 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:20:01.355 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509491 -lt 64424509492 2024-09-18T15:20:01.355 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:20:01.355 INFO:tasks.workunit.client.0.smithi046.stderr:27042one.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-64424509493 3-489626271747 4-107374182451 5-489626271747 6-150323855408 7-171798691887 8-193273528365 9-438086664204' 2024-09-18T15:20:01.355 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:20:01.355 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836534 2024-09-18T15:20:01.355 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:20:01.355 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:20:01.355 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836534 2024-09-18T15:20:01.356 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:20:01.356 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836534 2024-09-18T15:20:01.356 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2024-09-18T15:20:01.356 INFO:tasks.workunit.client.0.smithi046.stderr:270432: waiting osd.0 seq 21474836534 2024-09-18T15:20:01.356 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:20:01.356 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836534 -lt 21474836534 2024-09-18T15:20:01.356 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:20:01.356 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673012 2024-09-18T15:20:01.356 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:20:01.356 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:20:01.356 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673012 2024-09-18T15:20:01.356 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:20:01.356 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673012 2024-09-18T15:20:01.357 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673012' 2024-09-18T15:20:01.357 INFO:tasks.workunit.client.0.smithi046.stderr:270432: waiting osd.1 seq 42949673012 2024-09-18T15:20:01.357 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:20:01.357 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673013 -lt 42949673012 2024-09-18T15:20:01.357 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:20:01.357 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509493 2024-09-18T15:20:01.357 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:20:01.357 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:20:01.357 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509493 2024-09-18T15:20:01.357 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:20:01.357 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509493 2024-09-18T15:20:01.357 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509493' 2024-09-18T15:20:01.357 INFO:tasks.workunit.client.0.smithi046.stderr:270432: waiting osd.2 seq 64424509493 2024-09-18T15:20:01.358 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:20:01.358 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509491 -lt 64424509493 2024-09-18T15:20:01.358 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:20:02.684 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 8: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:20:02.685 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:20:02.685 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509494 -lt 64424509492 2024-09-18T15:20:02.685 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:20:02.685 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-489626271746 2024-09-18T15:20:02.685 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:20:02.685 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:20:02.685 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-489626271746 2024-09-18T15:20:02.685 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:20:02.685 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=489626271746 2024-09-18T15:20:02.685 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 489626271746' 2024-09-18T15:20:02.686 INFO:tasks.workunit.client.0.smithi046.stderr:270428: waiting osd.3 seq 489626271746 2024-09-18T15:20:02.686 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:20:02.686 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 489626271747 -lt 489626271746 2024-09-18T15:20:02.686 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:20:02.686 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182450 2024-09-18T15:20:02.686 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:20:02.686 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:20:02.686 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182450 2024-09-18T15:20:02.686 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:20:02.686 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182450 2024-09-18T15:20:02.686 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182450' 2024-09-18T15:20:02.686 INFO:tasks.workunit.client.0.smithi046.stderr:270428: waiting osd.4 seq 107374182450 2024-09-18T15:20:02.686 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:20:02.686 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182452 -lt 107374182450 2024-09-18T15:20:02.687 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:20:02.687 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-489626271746 2024-09-18T15:20:02.687 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:20:02.687 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:20:02.687 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:20:02.687 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-489626271746 2024-09-18T15:20:02.687 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=489626271746 2024-09-18T15:20:02.687 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 489626271746' 2024-09-18T15:20:02.687 INFO:tasks.workunit.client.0.smithi046.stderr:270428: waiting osd.5 seq 489626271746 2024-09-18T15:20:02.687 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:20:02.687 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 489626271748 -lt 489626271746 2024-09-18T15:20:02.701 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/q-eq 0 ']' 2024-09-18T15:20:02.702 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:20:02.702 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509494 -lt 64424509493 2024-09-18T15:20:02.702 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:20:02.702 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-489626271747 2024-09-18T15:20:02.702 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:20:02.702 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:20:02.702 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:20:02.702 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-489626271747 2024-09-18T15:20:02.702 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=489626271747 2024-09-18T15:20:02.702 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 489626271747' 2024-09-18T15:20:02.702 INFO:tasks.workunit.client.0.smithi046.stderr:270432: waiting osd.3 seq 489626271747 2024-09-18T15:20:02.703 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:20:02.703 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 489626271747 -lt 489626271747 2024-09-18T15:20:02.703 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:20:02.703 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182451 2024-09-18T15:20:02.703 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:20:02.703 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:20:02.703 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182451 2024-09-18T15:20:02.703 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:20:02.703 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182451 2024-09-18T15:20:02.703 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182451' 2024-09-18T15:20:02.703 INFO:tasks.workunit.client.0.smithi046.stderr:270432: waiting osd.4 seq 107374182451 2024-09-18T15:20:02.703 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:20:02.704 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182452 -lt 107374182451 2024-09-18T15:20:02.704 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:20:02.704 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-489626271747 2024-09-18T15:20:02.704 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:20:02.704 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:20:02.704 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-489626271747 2024-09-18T15:20:02.704 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:20:02.704 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=489626271747 2024-09-18T15:20:02.704 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 489626271747' 2024-09-18T15:20:02.704 INFO:tasks.workunit.client.0.smithi046.stderr:270432: waiting osd.5 seq 489626271747 2024-09-18T15:20:02.705 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:20:02.705 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 489626271748 -lt 489626271747 2024-09-18T15:20:02.705 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:20:03.770 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/cloa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:20:03.770 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855407 2024-09-18T15:20:03.770 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:20:03.770 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:20:03.770 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855407 2024-09-18T15:20:03.771 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:20:03.771 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855407 2024-09-18T15:20:03.771 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855407' 2024-09-18T15:20:03.771 INFO:tasks.workunit.client.0.smithi046.stderr:270428: waiting osd.6 seq 150323855407 2024-09-18T15:20:03.771 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:20:03.771 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855409 -lt 150323855407 2024-09-18T15:20:03.771 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:20:03.771 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691888 2024-09-18T15:20:03.771 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:20:03.771 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:20:03.771 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691888 2024-09-18T15:20:03.771 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:20:03.771 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691888 2024-09-18T15:20:03.771 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691888' 2024-09-18T15:20:03.772 INFO:tasks.workunit.client.0.smithi046.stderr:270428: waiting osd.7 seq 171798691888 2024-09-18T15:20:03.772 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:20:03.772 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691889 -lt 171798691888 2024-09-18T15:20:03.772 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:20:03.772 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528366 2024-09-18T15:20:03.772 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:20:03.772 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:20:03.772 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528366 2024-09-18T15:20:03.772 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:20:03.772 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528366 2024-09-18T15:20:03.772 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528366' 2024-09-18T15:20:03.772 INFO:tasks.workunit.client.0.smithi046.stderr:270428: waiting osd.8 seq 193273528366 2024-09-18T15:20:03.773 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:20:03.773 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528367 -lt 193273528366 2024-09-18T15:20:03.773 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:20:03.773 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-438086664205 2024-09-18T15:20:03.773 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:20:03.773 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:20:03.846 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /ne.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855408 2024-09-18T15:20:03.846 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:20:03.847 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:20:03.847 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855408 2024-09-18T15:20:03.847 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:20:03.847 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855408 2024-09-18T15:20:03.847 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855408' 2024-09-18T15:20:03.847 INFO:tasks.workunit.client.0.smithi046.stderr:270432: waiting osd.6 seq 150323855408 2024-09-18T15:20:03.847 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:20:03.847 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855409 -lt 150323855408 2024-09-18T15:20:03.847 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:20:03.847 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691887 2024-09-18T15:20:03.847 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:20:03.847 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:20:03.847 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691887 2024-09-18T15:20:03.848 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:20:03.848 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691887 2024-09-18T15:20:03.848 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691887' 2024-09-18T15:20:03.848 INFO:tasks.workunit.client.0.smithi046.stderr:270432: waiting osd.7 seq 171798691887 2024-09-18T15:20:03.848 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:20:03.848 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691889 -lt 171798691887 2024-09-18T15:20:03.848 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:20:03.848 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528365 2024-09-18T15:20:03.848 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:20:03.848 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:20:03.848 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528365 2024-09-18T15:20:03.848 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:20:03.849 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528365 2024-09-18T15:20:03.849 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528365' 2024-09-18T15:20:03.849 INFO:tasks.workunit.client.0.smithi046.stderr:270432: waiting osd.8 seq 193273528365 2024-09-18T15:20:03.849 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:20:03.849 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528367 -lt 193273528365 2024-09-18T15:20:03.849 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:20:03.849 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-438086664204 2024-09-18T15:20:03.849 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:20:03.849 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:20:05.261 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-4380/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-438086664205 2024-09-18T15:20:05.262 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:20:05.262 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=438086664205 2024-09-18T15:20:05.262 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 438086664205' 2024-09-18T15:20:05.262 INFO:tasks.workunit.client.0.smithi046.stderr:270428: waiting osd.9 seq 438086664205 2024-09-18T15:20:05.262 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:20:05.262 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 438086664205 -lt 438086664205 2024-09-18T15:20:05.262 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:20:05.262 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:20:05.262 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:20:05.262 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:20:05.262 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:20:05.263 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:20:05.263 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:20:05.263 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:20:05.263 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:20:05.263 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:20:05.263 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:20:05.263 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:20:05.263 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:20:05.263 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:20:05.263 INFO:tasks.workunit.client.0.smithi046.stderr:270428: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:20:05.263 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:20:05.263 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:20:05.263 INFO:tasks.workunit.client.0.smithi046.stderr:270428: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:20:05.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:20:05.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:20:05.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 270429 2024-09-18T15:20:05.278 INFO:tasks.workunit.client.0.smithi046.stderr:86664204 2024-09-18T15:20:05.279 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:20:05.279 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=438086664204 2024-09-18T15:20:05.279 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 438086664204' 2024-09-18T15:20:05.279 INFO:tasks.workunit.client.0.smithi046.stderr:270432: waiting osd.9 seq 438086664204 2024-09-18T15:20:05.279 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:20:05.279 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 438086664205 -lt 438086664204 2024-09-18T15:20:05.279 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:20:05.279 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:20:05.279 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:20:05.279 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:20:05.280 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:20:05.280 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:20:05.280 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:20:05.280 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:20:05.280 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:20:05.280 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:20:05.280 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:20:05.280 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:20:05.280 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:20:05.280 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:20:05.280 INFO:tasks.workunit.client.0.smithi046.stderr:270432: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:20:05.280 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:20:05.281 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:20:05.281 INFO:tasks.workunit.client.0.smithi046.stderr:270432: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:20:05.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:20:05.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: eval 'pids='\'''\''' 2024-09-18T15:20:05.282 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: pids= 2024-09-18T15:20:05.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: wait_background: return 0 2024-09-18T15:20:05.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: return_code=0 2024-09-18T15:20:05.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-18T15:20:05.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:214: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-18T15:20:05.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-18T15:20:05.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T15:20:05.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:20:05.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:20:05.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:20:05.376 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:20:05.377 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:20:05.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:20:05.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:20:05.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:20:05.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:20:05.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:20:05.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:20:05.515 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:20:05.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:20:05.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:20:05.518 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:20:05.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:20:05.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:20:05.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:20:05.521 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:20:05.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:20:05.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:20:05.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:20:05.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:20:05.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:20:05.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:20:05.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:20:05.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:20:05.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:20:05.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T15:20:05.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T15:20:05.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T15:20:05.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T15:20:05.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:20:05.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:20:05.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:20:05.595 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:20:05.595 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:20:05.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:20:05.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:20:05.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:20:05.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:20:05.600 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:20:05.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:20:05.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:20:05.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:20:05.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:20:05.606 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:20:05.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:20:05.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:20:05.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:20:05.609 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:20:05.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:20:05.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:20:05.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:20:05.616 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:20:05.616 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:20:05.616 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:20:05.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:20:05.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:20:05.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:20:05.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T15:20:05.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:20:05.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:20:05.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:20:05.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T15:20:05.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:20:05.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:20:05.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:20:05.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T15:20:05.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_corrupt_and_repair_lrc_overwrites td/osd-scrub-repair 2024-09-18T15:20:05.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:855: TEST_corrupt_and_repair_lrc_overwrites: '[' true = true ']' 2024-09-18T15:20:05.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:856: TEST_corrupt_and_repair_lrc_overwrites: corrupt_and_repair_lrc td/osd-scrub-repair true 2024-09-18T15:20:05.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-09-18T15:20:05.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: local allow_overwrites=true 2024-09-18T15:20:05.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: local poolname=ecpool 2024-09-18T15:20:05.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-09-18T15:20:05.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T15:20:05.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:20:05.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:20:05.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:20:05.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T15:20:05.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-18T15:20:06.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:20:06.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:20:06.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:20:06.073 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:20:06.073 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:20:06.073 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:20:06.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:20:06.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T15:20:06.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:20:06.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:20:06.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:20:06.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:20:06.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:20:06.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:20:06.356 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:20:06.356 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:20:06.356 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:20:06.357 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:20:06.357 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:20:06.357 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:20:06.358 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:20:06.359 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:20:06.359 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T15:20:06.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:20:06.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:20:06.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:20:06.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:20:06.454 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:20:06.454 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:20:06.454 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:20:06.454 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:20:06.454 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:20:06.455 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:20:06.455 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:20:06.455 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:20:06.455 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:20:06.456 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T15:20:06.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-09-18T15:20:06.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T15:20:06.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:20:06.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:20:06.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:20:06.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T15:20:06.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:20:07.420 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:20:07.420 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:20:07.420 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:20:07.421 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:20:07.421 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:20:07.421 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:20:07.421 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:20:07.422 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:20:07.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:20:07.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: seq 0 9 2024-09-18T15:20:07.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-18T15:20:07.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: '[' true = true ']' 2024-09-18T15:20:07.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-09-18T15:20:07.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:20:07.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:20:07.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:20:07.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:20:07.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:20:07.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:20:07.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:20:07.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:20:07.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:20:07.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:20:07.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:20:07.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:20:07.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:20:07.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:20:07.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:20:07.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:20:07.467 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:20:07.467 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:20:07.467 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:20:07.467 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:20:07.467 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:20:07.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:20:07.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:20:07.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:20:07.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:20:07.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:20:07.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:20:07.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:20:07.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:20:07.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:20:07.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:20:07.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:20:07.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:20:07.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:20:07.469 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:20:07.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9f5dc3f0-274f-4d64-8a0d-ec1dc42200b6 2024-09-18T15:20:07.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9f5dc3f0-274f-4d64-8a0d-ec1dc42200b6' 2024-09-18T15:20:07.469 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 9f5dc3f0-274f-4d64-8a0d-ec1dc42200b6 2024-09-18T15:20:07.469 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:20:07.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCn7+pmCW7DHBAAxO7owtV2fEn0eClKval/oQ== 2024-09-18T15:20:07.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCn7+pmCW7DHBAAxO7owtV2fEn0eClKval/oQ=="}' 2024-09-18T15:20:07.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9f5dc3f0-274f-4d64-8a0d-ec1dc42200b6 -i td/osd-scrub-repair/0/new.json 2024-09-18T15:20:07.779 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:20:07.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-18T15:20:07.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCn7+pmCW7DHBAAxO7owtV2fEn0eClKval/oQ== --osd-uuid 9f5dc3f0-274f-4d64-8a0d-ec1dc42200b6 2024-09-18T15:20:07.831 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:20:07.823+0000 7f472d421540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:20:07.873 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:20:07.863+0000 7f472d421540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:20:07.875 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:20:07.867+0000 7f472d421540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:20:07.876 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:20:07.867+0000 7f472d421540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:20:07.876 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:20:07.867+0000 7f472d421540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:20:07.876 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:20:07.867+0000 7f472d421540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:20:07.980 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:20:07.971+0000 7f472d421540 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-18T15:20:14.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T15:20:14.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:20:14.751 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T15:20:14.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:20:14.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:20:15.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:20:15.122 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T15:20:15.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:20:15.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:20:15.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:20:15.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:20:15.146 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:20:15.139+0000 7fbf6412b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:20:15.149 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:20:15.143+0000 7fbf6412b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:20:15.157 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:20:15.143+0000 7fbf6412b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:20:15.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:20:15.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:20:15.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:20:15.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:20:15.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:20:15.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:15.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:20:15.463 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:20:15.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:15.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:20:15.691 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:20:15.683+0000 7fbf6412b540 -1 Falling back to public interface 2024-09-18T15:20:15.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:16.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:16.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:16.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:20:16.766 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:20:16.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:16.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:20:17.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:18.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:18.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:18.086 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:20:18.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:20:18.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:18.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:20:18.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:18.523 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:20:18.515+0000 7fbf6412b540 -1 osd.0 0 log_to_monitors true 2024-09-18T15:20:19.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:19.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:19.398 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:20:19.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:20:19.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:19.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:20:19.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:20.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:20.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:20.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:20:20.760 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:20:20.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:20.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:20:21.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:22.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:22.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:22.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:20:22.080 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:20:22.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:22.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:20:22.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:23.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:23.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:23.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:20:23.448 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:20:23.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:23.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:20:23.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:24.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:24.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:24.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:20:24.767 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:20:24.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:24.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:20:25.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:26.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:26.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:26.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:20:26.083 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:20:26.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:26.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:20:26.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:27.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:27.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:27.423 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:20:27.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:20:27.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:27.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:20:27.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:28.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:28.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:28.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:20:28.739 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:20:28.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:28.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:20:29.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:30.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:30.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:30.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:20:30.057 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:20:30.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:30.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:20:30.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:31.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:31.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:31.442 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:20:31.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:20:31.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:31.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:20:31.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:32.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:32.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:32.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:20:32.997 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:20:32.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:32.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:20:33.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:34.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:34.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:34.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:20:34.642 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:20:34.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:34.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:20:35.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:36.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:36.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:36.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:20:36.468 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:20:36.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:36.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:20:37.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:38.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:38.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:38.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:20:38.135 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:20:38.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:38.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:20:38.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:39.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:39.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:39.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:20:39.454 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:20:39.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:39.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:20:39.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:40.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:40.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:40.904 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:20:40.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:20:40.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:40.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:20:41.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:42.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:42.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:42.550 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:20:42.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:20:42.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:42.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:20:42.926 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/273574,v1:127.0.0.1:6803/273574] [v2:127.0.0.1:6804/273574,v1:127.0.0.1:6805/273574] exists,up 9f5dc3f0-274f-4d64-8a0d-ec1dc42200b6 2024-09-18T15:20:42.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:20:42.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:20:42.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:20:42.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-18T15:20:42.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: '[' true = true ']' 2024-09-18T15:20:42.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-09-18T15:20:42.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:20:42.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:20:42.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:20:42.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:20:42.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:20:42.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:20:42.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:20:42.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:20:42.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:20:42.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:20:42.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:20:42.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:20:42.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:20:42.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:20:42.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:20:42.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:20:42.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:20:42.932 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:20:42.932 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:20:42.932 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:20:42.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:20:42.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:20:42.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:20:42.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:20:42.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:20:42.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:20:42.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:20:42.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:20:42.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:20:42.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:20:42.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:20:42.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:20:42.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:20:42.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:20:42.934 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:20:42.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=971a2b25-41fe-464f-9f92-077648d7a750 2024-09-18T15:20:42.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 971a2b25-41fe-464f-9f92-077648d7a750' 2024-09-18T15:20:42.935 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 971a2b25-41fe-464f-9f92-077648d7a750 2024-09-18T15:20:42.936 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:20:42.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDK7+pmqx6vOBAA45tOjFFx6j5ipgY3LJAULw== 2024-09-18T15:20:42.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDK7+pmqx6vOBAA45tOjFFx6j5ipgY3LJAULw=="}' 2024-09-18T15:20:42.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 971a2b25-41fe-464f-9f92-077648d7a750 -i td/osd-scrub-repair/1/new.json 2024-09-18T15:20:43.362 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:20:43.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-18T15:20:43.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDK7+pmqx6vOBAA45tOjFFx6j5ipgY3LJAULw== --osd-uuid 971a2b25-41fe-464f-9f92-077648d7a750 2024-09-18T15:20:43.414 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:20:43.406+0000 7f95c7375540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:20:43.419 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:20:43.410+0000 7f95c7375540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:20:43.421 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:20:43.414+0000 7f95c7375540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:20:43.422 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:20:43.414+0000 7f95c7375540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:20:43.422 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:20:43.414+0000 7f95c7375540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:20:43.422 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:20:43.414+0000 7f95c7375540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:20:43.486 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:20:43.478+0000 7f95c7375540 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-18T15:20:49.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-18T15:20:49.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:20:49.781 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T15:20:49.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:20:49.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:20:50.180 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T15:20:50.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:20:50.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:20:50.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:20:50.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:20:50.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:20:50.212 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:20:50.202+0000 7ffbcc04e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:20:50.237 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:20:50.230+0000 7ffbcc04e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:20:50.238 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:20:50.230+0000 7ffbcc04e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:20:50.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:20:50.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:20:50.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:20:50.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:20:50.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:20:50.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:50.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:20:50.522 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:20:50.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:50.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:20:50.787 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:20:50.778+0000 7ffbcc04e540 -1 Falling back to public interface 2024-09-18T15:20:50.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:51.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:51.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:51.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:20:51.897 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:20:51.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:51.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:20:52.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:53.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:53.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:53.224 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:20:53.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:20:53.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:53.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:20:53.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:53.600 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:20:53.590+0000 7ffbcc04e540 -1 osd.1 0 log_to_monitors true 2024-09-18T15:20:54.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:54.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:54.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:20:54.569 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:20:54.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:54.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:20:54.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:55.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:55.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:55.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:20:55.934 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:20:55.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:55.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:20:56.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:57.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:57.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:57.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:20:57.278 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:20:57.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:57.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:20:57.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:58.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:58.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:58.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:20:58.815 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:20:58.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:58.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:20:59.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:00.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:00.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:00.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:21:00.128 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:21:00.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:00.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:21:00.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:01.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:01.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:01.494 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:21:01.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:21:01.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:01.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:21:01.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:02.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:02.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:02.903 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:21:02.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:21:02.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:21:02.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:03.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:04.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:04.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:04.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:21:04.217 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:21:04.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:04.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:21:04.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:05.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:05.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:05.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:21:05.672 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:21:05.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:05.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:21:06.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:07.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:07.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:07.072 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:21:07.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:21:07.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:07.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:21:07.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:08.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:08.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:08.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:21:08.482 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:21:08.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:08.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:21:09.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:10.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:10.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:10.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:21:10.093 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:21:10.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:10.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:21:10.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:11.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:11.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:11.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:21:11.531 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:21:11.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:11.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:21:11.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:12.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:12.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:12.926 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:21:12.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:21:12.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:12.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:21:13.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:14.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:14.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:14.485 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:21:14.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:21:14.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:14.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:21:14.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:15.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:15.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:15.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:21:15.993 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:21:15.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:15.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:21:16.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:17.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:17.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:17.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:21:17.384 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:21:17.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:17.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:21:17.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:18.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:18.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:18.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-18T15:21:18.902 INFO:tasks.workunit.client.0.smithi046.stdout:20 2024-09-18T15:21:18.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:18.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:21:19.229 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/275440,v1:127.0.0.1:6811/275440] [v2:127.0.0.1:6812/275440,v1:127.0.0.1:6813/275440] exists,up 971a2b25-41fe-464f-9f92-077648d7a750 2024-09-18T15:21:19.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:21:19.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:21:19.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:21:19.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-18T15:21:19.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: '[' true = true ']' 2024-09-18T15:21:19.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-09-18T15:21:19.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:21:19.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:21:19.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T15:21:19.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:21:19.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:21:19.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:21:19.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:21:19.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:21:19.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:21:19.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T15:21:19.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T15:21:19.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:21:19.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:21:19.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:21:19.234 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:21:19.234 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:21:19.234 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:21:19.234 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:21:19.234 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:21:19.234 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:21:19.236 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:21:19.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:21:19.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:21:19.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:21:19.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:21:19.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:21:19.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:21:19.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:21:19.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:21:19.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:21:19.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:21:19.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:21:19.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:21:19.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-18T15:21:19.237 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:21:19.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a3e9d66b-9ef0-400a-a0bb-66760d2c5ada 2024-09-18T15:21:19.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 a3e9d66b-9ef0-400a-a0bb-66760d2c5ada' 2024-09-18T15:21:19.238 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 a3e9d66b-9ef0-400a-a0bb-66760d2c5ada 2024-09-18T15:21:19.238 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:21:19.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDv7+pmkRgPDxAAcIs0H6Lc4ODTbxiNfWQafg== 2024-09-18T15:21:19.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDv7+pmkRgPDxAAcIs0H6Lc4ODTbxiNfWQafg=="}' 2024-09-18T15:21:19.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a3e9d66b-9ef0-400a-a0bb-66760d2c5ada -i td/osd-scrub-repair/2/new.json 2024-09-18T15:21:19.711 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:21:19.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-18T15:21:19.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDv7+pmkRgPDxAAcIs0H6Lc4ODTbxiNfWQafg== --osd-uuid a3e9d66b-9ef0-400a-a0bb-66760d2c5ada 2024-09-18T15:21:19.763 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:21:19.754+0000 7f12fe6e8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:21:19.818 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:21:19.810+0000 7f12fe6e8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:21:19.820 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:21:19.810+0000 7f12fe6e8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:21:19.821 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:21:19.814+0000 7f12fe6e8540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T15:21:19.822 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:21:19.814+0000 7f12fe6e8540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T15:21:19.822 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:21:19.814+0000 7f12fe6e8540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T15:21:19.891 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:21:19.882+0000 7f12fe6e8540 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-18T15:21:26.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-18T15:21:26.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:21:26.196 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T15:21:26.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T15:21:26.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:21:26.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T15:21:26.583 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T15:21:26.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:21:26.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:21:26.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:21:26.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:21:26.614 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:21:26.606+0000 7f2df3412540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:21:26.617 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:21:26.610+0000 7f2df3412540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:21:26.618 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:21:26.610+0000 7f2df3412540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:21:26.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T15:21:26.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:21:26.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:21:26.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:21:26.911 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:21:26.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:21:26.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:26.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:21:26.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:26.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:21:27.155 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:21:27.146+0000 7f2df3412540 -1 Falling back to public interface 2024-09-18T15:21:27.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:28.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:28.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:28.255 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:21:28.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:21:28.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:28.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:21:28.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:29.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:29.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:29.572 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:21:29.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:21:29.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:29.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:21:29.725 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:21:29.714+0000 7f2df3412540 -1 osd.2 0 log_to_monitors true 2024-09-18T15:21:29.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:30.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:30.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:30.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:21:30.963 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:21:30.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:30.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:21:31.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:32.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:32.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:32.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:21:32.281 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:21:32.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:32.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:21:32.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:33.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:33.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:33.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:21:33.599 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:21:33.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:33.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:21:33.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:34.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:34.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:34.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:21:34.934 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:21:34.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:34.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:21:35.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:36.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:36.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:36.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:21:36.259 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:21:36.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:36.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:21:36.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:37.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:37.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:37.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:21:37.583 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:21:37.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:37.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:21:38.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:39.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:39.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:39.039 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:21:39.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:21:39.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:39.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:21:39.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:40.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:40.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:40.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:21:40.397 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:21:40.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:40.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:21:40.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:41.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:41.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:41.849 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:21:41.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:21:41.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:41.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:21:42.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:43.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:43.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:43.221 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:21:43.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:21:43.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:43.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:21:43.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:44.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:44.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:44.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:21:44.921 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:21:44.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:44.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:21:45.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:46.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:46.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:46.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:21:46.526 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:21:46.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:46.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:21:47.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:48.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:48.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:48.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:21:48.335 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:21:48.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:48.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:21:49.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:50.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:50.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:50.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:21:50.308 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:21:50.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:50.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:21:50.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:51.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:51.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:51.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:21:51.974 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:21:51.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:51.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:21:52.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:53.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:53.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:53.295 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:21:53.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:21:53.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:53.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:21:53.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:54.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:54.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:54.990 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:21:54.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:21:54.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:54.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:21:55.310 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/277270,v1:127.0.0.1:6819/277270] [v2:127.0.0.1:6820/277270,v1:127.0.0.1:6821/277270] exists,up a3e9d66b-9ef0-400a-a0bb-66760d2c5ada 2024-09-18T15:21:55.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:21:55.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:21:55.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:21:55.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-18T15:21:55.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: '[' true = true ']' 2024-09-18T15:21:55.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-09-18T15:21:55.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:21:55.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:21:55.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T15:21:55.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:21:55.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:21:55.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:21:55.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:21:55.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:21:55.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:21:55.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-18T15:21:55.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-18T15:21:55.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:21:55.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:21:55.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:21:55.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:21:55.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:21:55.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:21:55.316 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:21:55.316 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:21:55.316 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:21:55.316 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:21:55.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:21:55.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:21:55.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:21:55.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:21:55.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:21:55.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:21:55.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:21:55.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:21:55.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:21:55.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:21:55.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:21:55.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:21:55.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-09-18T15:21:55.318 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:21:55.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=105fcb65-1d2b-4358-9b52-7d1b65a95f69 2024-09-18T15:21:55.319 INFO:tasks.workunit.client.0.smithi046.stdout:add osd3 105fcb65-1d2b-4358-9b52-7d1b65a95f69 2024-09-18T15:21:55.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 105fcb65-1d2b-4358-9b52-7d1b65a95f69' 2024-09-18T15:21:55.320 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:21:55.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAT8OpmBZ+/ExAAQuenublZZHPhipMzAnzzLQ== 2024-09-18T15:21:55.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAT8OpmBZ+/ExAAQuenublZZHPhipMzAnzzLQ=="}' 2024-09-18T15:21:55.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 105fcb65-1d2b-4358-9b52-7d1b65a95f69 -i td/osd-scrub-repair/3/new.json 2024-09-18T15:21:55.731 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:21:55.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-09-18T15:21:55.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAT8OpmBZ+/ExAAQuenublZZHPhipMzAnzzLQ== --osd-uuid 105fcb65-1d2b-4358-9b52-7d1b65a95f69 2024-09-18T15:21:55.780 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:21:55.769+0000 7fc4c973c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:21:55.797 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:21:55.785+0000 7fc4c973c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:21:55.798 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:21:55.789+0000 7fc4c973c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:21:55.799 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:21:55.789+0000 7fc4c973c540 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-09-18T15:21:55.799 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:21:55.789+0000 7fc4c973c540 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-09-18T15:21:55.799 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:21:55.789+0000 7fc4c973c540 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-09-18T15:21:55.871 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:21:55.861+0000 7fc4c973c540 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-09-18T15:22:02.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-18T15:22:02.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:22:02.264 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd3 key to auth repository 2024-09-18T15:22:02.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T15:22:02.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:22:02.660 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.3 2024-09-18T15:22:02.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T15:22:02.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:22:02.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:22:02.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:22:02.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:22:02.691 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:22:02.681+0000 7f3032514540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:22:02.694 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:22:02.685+0000 7f3032514540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:22:02.695 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:22:02.685+0000 7f3032514540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:22:02.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T15:22:02.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:22:02.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:22:02.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:22:02.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:22:02.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:02.958 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:22:02.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:22:02.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:02.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:22:03.236 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:22:03.225+0000 7f3032514540 -1 Falling back to public interface 2024-09-18T15:22:03.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:04.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:04.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:04.264 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:22:04.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:22:04.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:04.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:22:04.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:05.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:05.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:05.583 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:22:05.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:22:05.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:05.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:22:05.989 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:22:05.981+0000 7f3032514540 -1 osd.3 0 log_to_monitors true 2024-09-18T15:22:06.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:07.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:07.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:07.012 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:22:07.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:22:07.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:07.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:22:07.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:08.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:08.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:08.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:22:08.352 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:22:08.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:08.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:22:08.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:09.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:09.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:09.665 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:22:09.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:22:09.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:09.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:22:10.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:11.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:11.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:11.005 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:22:11.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:22:11.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:11.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:22:11.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:12.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:12.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:12.317 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:22:12.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:22:12.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:12.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:22:12.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:13.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:13.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:13.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:22:13.650 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:22:13.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:13.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:22:14.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:15.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:15.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:15.029 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:22:15.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:22:15.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:22:15.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:15.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:16.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:16.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:16.346 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:22:16.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:22:16.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:16.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:22:16.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:17.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:17.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:17.666 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:22:17.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:22:17.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:17.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:22:17.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:18.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:18.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:18.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:22:18.990 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:22:18.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:18.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:22:19.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:20.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:20.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:20.358 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:22:20.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:22:20.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:20.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:22:21.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:22.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:22.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:22.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:22:22.087 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:22:22.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:22.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:22:22.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:23.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:23.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:23.586 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:22:23.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:22:23.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:23.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:22:24.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:25.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:25.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:25.016 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:22:25.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:22:25.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:25.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:22:25.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:26.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:26.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:26.447 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:22:26.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:22:26.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:26.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:22:26.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:27.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:27.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:27.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:22:27.907 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:22:27.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:27.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:22:28.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:29.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:29.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:29.711 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:22:29.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:22:29.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:29.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:22:30.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:31.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:31.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:31.158 INFO:tasks.workunit.client.0.smithi046.stdout:20 2024-09-18T15:22:31.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-18T15:22:31.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:31.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:22:31.493 INFO:tasks.workunit.client.0.smithi046.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/279074,v1:127.0.0.1:6827/279074] [v2:127.0.0.1:6828/279074,v1:127.0.0.1:6829/279074] exists,up 105fcb65-1d2b-4358-9b52-7d1b65a95f69 2024-09-18T15:22:31.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:22:31.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:22:31.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:22:31.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-18T15:22:31.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: '[' true = true ']' 2024-09-18T15:22:31.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-09-18T15:22:31.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:22:31.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:22:31.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-18T15:22:31.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:22:31.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/4 2024-09-18T15:22:31.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:22:31.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:22:31.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:22:31.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:22:31.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-09-18T15:22:31.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-09-18T15:22:31.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:22:31.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:22:31.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:22:31.499 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:22:31.499 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:22:31.499 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:22:31.499 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:22:31.499 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:22:31.499 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:22:31.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:22:31.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:22:31.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:22:31.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:22:31.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:22:31.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:22:31.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:22:31.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:22:31.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:22:31.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:22:31.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:22:31.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:22:31.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:22:31.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/4 2024-09-18T15:22:31.501 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:22:31.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=49384905-ad5e-40ee-92fc-e185e4488241 2024-09-18T15:22:31.502 INFO:tasks.workunit.client.0.smithi046.stdout:add osd4 49384905-ad5e-40ee-92fc-e185e4488241 2024-09-18T15:22:31.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 49384905-ad5e-40ee-92fc-e185e4488241' 2024-09-18T15:22:31.503 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:22:31.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA38OpmoKMGHxAAcBLxWNzW1KYQkmExPdtBWQ== 2024-09-18T15:22:31.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA38OpmoKMGHxAAcBLxWNzW1KYQkmExPdtBWQ=="}' 2024-09-18T15:22:31.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 49384905-ad5e-40ee-92fc-e185e4488241 -i td/osd-scrub-repair/4/new.json 2024-09-18T15:22:31.929 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:22:31.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/4/new.json 2024-09-18T15:22:31.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA38OpmoKMGHxAAcBLxWNzW1KYQkmExPdtBWQ== --osd-uuid 49384905-ad5e-40ee-92fc-e185e4488241 2024-09-18T15:22:31.981 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:22:31.973+0000 7f273c0c7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:22:31.986 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:22:31.977+0000 7f273c0c7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:22:31.988 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:22:31.977+0000 7f273c0c7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:22:31.989 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:22:31.981+0000 7f273c0c7540 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-09-18T15:22:31.989 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:22:31.981+0000 7f273c0c7540 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-09-18T15:22:31.990 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:22:31.981+0000 7f273c0c7540 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-09-18T15:22:32.057 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:22:32.049+0000 7f273c0c7540 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-09-18T15:22:38.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-09-18T15:22:38.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:22:38.736 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd4 key to auth repository 2024-09-18T15:22:38.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-18T15:22:38.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:22:39.126 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.4 2024-09-18T15:22:39.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-18T15:22:39.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:22:39.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:22:39.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:22:39.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:22:39.156 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:22:39.145+0000 7f6b37cbf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:22:39.158 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:22:39.149+0000 7f6b37cbf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:22:39.161 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:22:39.149+0000 7f6b37cbf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:22:39.448 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:22:39.437+0000 7f6b37cbf540 -1 Falling back to public interface 2024-09-18T15:22:39.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-18T15:22:39.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:22:39.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-18T15:22:39.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:22:39.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:22:39.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:39.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:22:39.513 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:22:39.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:39.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:22:39.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:40.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:40.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:40.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:22:40.847 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:22:40.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:40.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:22:41.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:42.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:42.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:42.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:22:42.156 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:22:42.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:42.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:22:42.326 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:22:42.317+0000 7f6b37cbf540 -1 osd.4 0 log_to_monitors true 2024-09-18T15:22:42.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:43.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:43.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:43.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:22:43.542 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:22:43.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:43.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:22:43.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:44.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:44.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:44.863 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:22:44.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:22:44.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:44.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:22:45.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:46.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:46.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:46.231 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:22:46.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:22:46.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:46.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:22:46.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:47.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:47.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:47.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:22:47.553 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:22:47.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:47.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:22:47.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:48.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:48.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:48.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:22:48.998 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:22:48.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:48.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:22:49.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:50.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:50.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:50.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:22:50.458 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:22:50.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:50.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:22:50.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:51.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:51.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:51.770 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:22:51.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:22:51.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:51.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:22:52.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:53.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:53.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:53.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:22:53.223 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:22:53.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:53.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:22:53.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:54.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:54.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:54.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:22:54.545 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:22:54.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:54.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:22:54.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:55.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:55.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:55.867 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:22:55.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:22:55.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:22:55.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:56.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:57.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:57.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:57.458 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:22:57.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:22:57.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:22:57.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:57.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:58.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:58.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:58.854 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:22:58.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:22:58.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:58.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:22:59.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:00.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:00.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:00.340 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:23:00.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:23:00.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:00.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:23:00.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:01.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:01.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:01.896 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:23:01.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:23:01.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:01.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:23:02.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:03.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:03.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:03.217 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:23:03.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:23:03.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:23:03.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:03.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:04.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:04.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:04.930 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:23:04.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:23:04.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:04.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:23:05.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:06.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:06.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:06.441 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:23:06.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:23:06.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:06.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:23:06.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:07.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:07.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:07.774 INFO:tasks.workunit.client.0.smithi046.stdout:20 2024-09-18T15:23:07.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-18T15:23:07.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:07.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:23:08.152 INFO:tasks.workunit.client.0.smithi046.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/280903,v1:127.0.0.1:6835/280903] [v2:127.0.0.1:6836/280903,v1:127.0.0.1:6837/280903] exists,up 49384905-ad5e-40ee-92fc-e185e4488241 2024-09-18T15:23:08.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:23:08.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:23:08.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:23:08.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-18T15:23:08.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: '[' true = true ']' 2024-09-18T15:23:08.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-09-18T15:23:08.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:23:08.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:23:08.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-18T15:23:08.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:23:08.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/5 2024-09-18T15:23:08.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:23:08.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:23:08.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:23:08.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:23:08.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-18T15:23:08.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-18T15:23:08.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:23:08.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:23:08.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:23:08.156 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:23:08.156 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:23:08.156 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:23:08.158 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:23:08.158 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:23:08.158 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:23:08.158 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:23:08.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:23:08.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:23:08.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:23:08.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:23:08.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:23:08.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:23:08.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:23:08.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:23:08.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:23:08.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:23:08.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:23:08.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:23:08.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/5 2024-09-18T15:23:08.159 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:23:08.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=63244125-a488-402d-9bdf-d4f6f2b1fbf0 2024-09-18T15:23:08.160 INFO:tasks.workunit.client.0.smithi046.stdout:add osd5 63244125-a488-402d-9bdf-d4f6f2b1fbf0 2024-09-18T15:23:08.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 63244125-a488-402d-9bdf-d4f6f2b1fbf0' 2024-09-18T15:23:08.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:23:08.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBc8OpmHY9xChAAHY2xI+EomCCrD6LwRUlK4A== 2024-09-18T15:23:08.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBc8OpmHY9xChAAHY2xI+EomCCrD6LwRUlK4A=="}' 2024-09-18T15:23:08.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 63244125-a488-402d-9bdf-d4f6f2b1fbf0 -i td/osd-scrub-repair/5/new.json 2024-09-18T15:23:08.637 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:23:08.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/5/new.json 2024-09-18T15:23:08.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBc8OpmHY9xChAAHY2xI+EomCCrD6LwRUlK4A== --osd-uuid 63244125-a488-402d-9bdf-d4f6f2b1fbf0 2024-09-18T15:23:08.690 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:23:08.680+0000 7f0b8a1cb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:23:08.703 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:23:08.692+0000 7f0b8a1cb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:23:08.705 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:23:08.696+0000 7f0b8a1cb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:23:08.706 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:23:08.696+0000 7f0b8a1cb540 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-09-18T15:23:08.706 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:23:08.696+0000 7f0b8a1cb540 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-09-18T15:23:08.706 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:23:08.696+0000 7f0b8a1cb540 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-09-18T15:23:08.768 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:23:08.756+0000 7f0b8a1cb540 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-09-18T15:23:15.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-09-18T15:23:15.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:23:15.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-18T15:23:15.333 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd5 key to auth repository 2024-09-18T15:23:15.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:23:15.718 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.5 2024-09-18T15:23:15.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-18T15:23:15.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:23:15.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:23:15.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:23:15.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:23:15.748 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:23:15.736+0000 7fe74af32540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:23:15.751 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:23:15.740+0000 7fe74af32540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:23:15.753 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:23:15.740+0000 7fe74af32540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:23:16.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-18T15:23:16.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:23:16.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T15:23:16.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:23:16.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:23:16.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:16.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:23:16.037 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:23:16.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:16.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:23:16.296 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:23:16.284+0000 7fe74af32540 -1 Falling back to public interface 2024-09-18T15:23:16.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:17.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:17.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:17.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:23:17.379 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:23:17.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:17.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:23:17.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:18.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:18.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:18.686 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:23:18.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:23:18.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:18.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:23:18.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:19.214 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:23:19.204+0000 7fe74af32540 -1 osd.5 0 log_to_monitors true 2024-09-18T15:23:19.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:19.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:19.995 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:23:19.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:23:19.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:19.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:23:20.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:21.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:21.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:21.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:23:21.333 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:23:21.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:21.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:23:21.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:22.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:22.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:22.655 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:23:22.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:23:22.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:22.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:23:23.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:24.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:24.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:24.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:23:24.136 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:23:24.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:24.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:23:24.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:25.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:25.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:25.447 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:23:25.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:23:25.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:25.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:23:25.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:26.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:26.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:26.890 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:23:26.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:23:26.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:26.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:23:27.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:28.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:28.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:28.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:23:28.339 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:23:28.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:28.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:23:28.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:29.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:29.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:29.658 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:23:29.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:23:29.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:29.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:23:29.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:30.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:30.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:30.985 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:23:30.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:23:30.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:30.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:23:31.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:32.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:32.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:32.418 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:23:32.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:23:32.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:32.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:23:32.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:33.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:33.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:33.746 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:23:33.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:23:33.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:33.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:23:34.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:35.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:35.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:35.308 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:23:35.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:23:35.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:23:35.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:35.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:36.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:36.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:36.926 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:23:36.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:23:36.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:36.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:23:37.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:38.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:38.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:38.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:23:38.418 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:23:38.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:38.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:23:39.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:40.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:40.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:40.242 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:23:40.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:23:40.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:40.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:23:40.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:41.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:41.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:41.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:23:41.612 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:23:41.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:41.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:23:42.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:43.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:43.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:43.084 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:23:43.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:23:43.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:43.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:23:43.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:44.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:44.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:44.555 INFO:tasks.workunit.client.0.smithi046.stdout:20 2024-09-18T15:23:44.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-18T15:23:44.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:44.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:23:44.864 INFO:tasks.workunit.client.0.smithi046.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/282732,v1:127.0.0.1:6843/282732] [v2:127.0.0.1:6844/282732,v1:127.0.0.1:6845/282732] exists,up 63244125-a488-402d-9bdf-d4f6f2b1fbf0 2024-09-18T15:23:44.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:23:44.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:23:44.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:23:44.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-18T15:23:44.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: '[' true = true ']' 2024-09-18T15:23:44.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-09-18T15:23:44.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:23:44.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:23:44.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-09-18T15:23:44.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:23:44.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/6 2024-09-18T15:23:44.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:23:44.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:23:44.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:23:44.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:23:44.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-09-18T15:23:44.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-09-18T15:23:44.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:23:44.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:23:44.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:23:44.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:23:44.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:23:44.868 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:23:44.870 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:23:44.870 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:23:44.870 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:23:44.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:23:44.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:23:44.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:23:44.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:23:44.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:23:44.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:23:44.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:23:44.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:23:44.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:23:44.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:23:44.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:23:44.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:23:44.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:23:44.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/6 2024-09-18T15:23:44.871 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:23:44.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c445ea25-e04f-4ffb-9eb8-605ab64e49c0 2024-09-18T15:23:44.872 INFO:tasks.workunit.client.0.smithi046.stdout:add osd6 c445ea25-e04f-4ffb-9eb8-605ab64e49c0 2024-09-18T15:23:44.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 c445ea25-e04f-4ffb-9eb8-605ab64e49c0' 2024-09-18T15:23:44.873 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:23:44.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCA8Opm/mXfNBAApeKlw9uoKfL5lUKn4k7iVQ== 2024-09-18T15:23:44.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCA8Opm/mXfNBAApeKlw9uoKfL5lUKn4k7iVQ=="}' 2024-09-18T15:23:44.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c445ea25-e04f-4ffb-9eb8-605ab64e49c0 -i td/osd-scrub-repair/6/new.json 2024-09-18T15:23:45.356 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:23:45.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/6/new.json 2024-09-18T15:23:45.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCA8Opm/mXfNBAApeKlw9uoKfL5lUKn4k7iVQ== --osd-uuid c445ea25-e04f-4ffb-9eb8-605ab64e49c0 2024-09-18T15:23:45.413 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:23:45.408+0000 7fbc1780d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:23:45.551 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:23:45.544+0000 7fbc1780d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:23:45.553 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:23:45.548+0000 7fbc1780d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:23:45.555 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:23:45.548+0000 7fbc1780d540 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-09-18T15:23:45.555 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:23:45.548+0000 7fbc1780d540 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-09-18T15:23:45.555 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:23:45.548+0000 7fbc1780d540 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-09-18T15:23:45.605 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:23:45.596+0000 7fbc1780d540 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-09-18T15:23:52.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-09-18T15:23:52.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:23:52.216 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd6 key to auth repository 2024-09-18T15:23:52.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-09-18T15:23:52.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:23:52.603 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.6 2024-09-18T15:23:52.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-09-18T15:23:52.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:23:52.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:23:52.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:23:52.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:23:52.633 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:23:52.628+0000 7fb9c40a5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:23:52.636 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:23:52.628+0000 7fb9c40a5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:23:52.640 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:23:52.632+0000 7fb9c40a5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:23:52.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-09-18T15:23:52.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:23:52.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-09-18T15:23:52.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:23:52.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:23:52.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:52.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:23:52.966 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:23:52.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:52.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T15:23:53.192 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:23:53.184+0000 7fb9c40a5540 -1 Falling back to public interface 2024-09-18T15:23:53.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:54.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:54.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:54.335 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:23:54.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:23:54.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:54.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T15:23:54.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:55.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:55.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:55.647 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:23:55.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:23:55.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:55.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T15:23:55.842 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:23:55.835+0000 7fb9c40a5540 -1 osd.6 0 log_to_monitors true 2024-09-18T15:23:56.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:57.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:57.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:57.071 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:23:57.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:23:57.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:57.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T15:23:57.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:58.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:58.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:58.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:23:58.385 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:23:58.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:58.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T15:23:58.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:59.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:59.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:59.708 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:23:59.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:23:59.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:59.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T15:24:00.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:01.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:01.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:01.061 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:24:01.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:24:01.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:01.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T15:24:01.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:02.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:02.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:02.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:24:02.384 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:24:02.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:02.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T15:24:02.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:03.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:03.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:03.770 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:24:03.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:24:03.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:03.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T15:24:04.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:05.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:05.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:05.103 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:24:05.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:24:05.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:05.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T15:24:05.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:06.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:06.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:06.427 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:24:06.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:24:06.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:06.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T15:24:06.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:07.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:07.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:07.743 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:24:07.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:24:07.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:07.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T15:24:08.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:09.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:09.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:09.097 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:24:09.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:24:09.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:09.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T15:24:09.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:10.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:10.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:10.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:24:10.409 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:24:10.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:10.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T15:24:10.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:11.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:11.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:11.962 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:24:11.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:24:11.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:11.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T15:24:12.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:13.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:13.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:13.374 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:24:13.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:24:13.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:13.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T15:24:13.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:14.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:14.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:14.752 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:24:14.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:24:14.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:14.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T15:24:15.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:16.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:16.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:16.278 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:24:16.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:24:16.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:16.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T15:24:16.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:17.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:17.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:17.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:24:17.817 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:24:17.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:17.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T15:24:18.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:19.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:19.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:19.345 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:24:19.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:24:19.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:19.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T15:24:19.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:20.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:20.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:20.841 INFO:tasks.workunit.client.0.smithi046.stdout:20 2024-09-18T15:24:20.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-18T15:24:20.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:20.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T15:24:21.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:22.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:22.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:22.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-18T15:24:22.260 INFO:tasks.workunit.client.0.smithi046.stdout:21 2024-09-18T15:24:22.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:22.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T15:24:22.563 INFO:tasks.workunit.client.0.smithi046.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/284561,v1:127.0.0.1:6851/284561] [v2:127.0.0.1:6852/284561,v1:127.0.0.1:6853/284561] exists,up c445ea25-e04f-4ffb-9eb8-605ab64e49c0 2024-09-18T15:24:22.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:24:22.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:24:22.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:24:22.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-18T15:24:22.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: '[' true = true ']' 2024-09-18T15:24:22.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-09-18T15:24:22.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:24:22.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:24:22.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-09-18T15:24:22.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:24:22.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/7 2024-09-18T15:24:22.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:24:22.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:24:22.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:24:22.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:24:22.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-09-18T15:24:22.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-09-18T15:24:22.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:24:22.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:24:22.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:24:22.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:24:22.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:24:22.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:24:22.569 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:24:22.569 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:24:22.569 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:24:22.569 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:24:22.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:24:22.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:24:22.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:24:22.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:24:22.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:24:22.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:24:22.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:24:22.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:24:22.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:24:22.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:24:22.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:24:22.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:24:22.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/7 2024-09-18T15:24:22.570 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:24:22.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6aa9d197-9465-4e12-bbc7-80e0393720e0 2024-09-18T15:24:22.572 INFO:tasks.workunit.client.0.smithi046.stdout:add osd7 6aa9d197-9465-4e12-bbc7-80e0393720e0 2024-09-18T15:24:22.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 6aa9d197-9465-4e12-bbc7-80e0393720e0' 2024-09-18T15:24:22.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:24:22.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCm8OpmYToHIxAA+4MLBsBHPExApTdmB3tS1Q== 2024-09-18T15:24:22.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCm8OpmYToHIxAA+4MLBsBHPExApTdmB3tS1Q=="}' 2024-09-18T15:24:22.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6aa9d197-9465-4e12-bbc7-80e0393720e0 -i td/osd-scrub-repair/7/new.json 2024-09-18T15:24:23.122 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:24:23.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/7/new.json 2024-09-18T15:24:23.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCm8OpmYToHIxAA+4MLBsBHPExApTdmB3tS1Q== --osd-uuid 6aa9d197-9465-4e12-bbc7-80e0393720e0 2024-09-18T15:24:23.178 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:24:23.171+0000 7ff98318b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:24:23.299 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:24:23.291+0000 7ff98318b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:24:23.301 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:24:23.295+0000 7ff98318b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:24:23.303 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:24:23.295+0000 7ff98318b540 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-09-18T15:24:23.303 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:24:23.295+0000 7ff98318b540 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-09-18T15:24:23.303 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:24:23.295+0000 7ff98318b540 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-09-18T15:24:23.411 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:24:23.403+0000 7ff98318b540 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-09-18T15:24:29.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-09-18T15:24:29.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:24:29.956 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd7 key to auth repository 2024-09-18T15:24:29.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-09-18T15:24:29.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:24:30.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-09-18T15:24:30.342 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.7 2024-09-18T15:24:30.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:24:30.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:24:30.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:24:30.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:24:30.373 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:24:30.363+0000 7f669adaf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:24:30.375 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:24:30.367+0000 7f669adaf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:24:30.381 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:24:30.367+0000 7f669adaf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:24:30.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-09-18T15:24:30.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:24:30.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-09-18T15:24:30.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:24:30.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:24:30.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:30.638 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:24:30.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:24:30.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:30.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T15:24:30.924 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:24:30.915+0000 7f669adaf540 -1 Falling back to public interface 2024-09-18T15:24:30.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:31.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:31.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:31.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:24:31.952 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:24:31.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:31.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T15:24:32.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:33.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:33.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:33.264 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:24:33.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:24:33.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:33.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T15:24:33.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:33.827 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:24:33.819+0000 7f669adaf540 -1 osd.7 0 log_to_monitors true 2024-09-18T15:24:34.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:34.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:34.576 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:24:34.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:24:34.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:34.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T15:24:34.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:35.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:35.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:35.888 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:24:35.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:24:35.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:35.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T15:24:36.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:37.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:37.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:37.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:24:37.274 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:24:37.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:37.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T15:24:37.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:38.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:38.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:38.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:24:38.655 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:24:38.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:38.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T15:24:38.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:39.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:39.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:39.981 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:24:39.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:24:39.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:39.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T15:24:40.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:41.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:41.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:41.295 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:24:41.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:24:41.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:41.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T15:24:41.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:42.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:42.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:42.689 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:24:42.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:24:42.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:42.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T15:24:43.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:44.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:44.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:44.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:24:44.022 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:24:44.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:44.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T15:24:44.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:45.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:45.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:45.395 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:24:45.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:24:45.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:45.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T15:24:45.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:46.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:46.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:46.714 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:24:46.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:24:46.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:46.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T15:24:47.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:48.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:48.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:48.256 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:24:48.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:24:48.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:48.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T15:24:48.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:49.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:49.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:49.808 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:24:49.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:24:49.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:49.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T15:24:50.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:51.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:51.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:51.199 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:24:51.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:24:51.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:51.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T15:24:51.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:52.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:52.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:52.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:24:52.664 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:24:52.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:52.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T15:24:53.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:54.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:54.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:54.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:24:54.102 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:24:54.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:54.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T15:24:54.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:55.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:55.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:55.952 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:24:55.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:24:55.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:55.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T15:24:56.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:57.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:57.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:57.334 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:24:57.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:24:57.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:57.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T15:24:57.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:58.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:58.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:58.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-18T15:24:58.814 INFO:tasks.workunit.client.0.smithi046.stdout:20 2024-09-18T15:24:58.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:58.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T15:24:59.150 INFO:tasks.workunit.client.0.smithi046.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/286415,v1:127.0.0.1:6859/286415] [v2:127.0.0.1:6860/286415,v1:127.0.0.1:6861/286415] exists,up 6aa9d197-9465-4e12-bbc7-80e0393720e0 2024-09-18T15:24:59.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:24:59.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:24:59.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:24:59.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-18T15:24:59.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: '[' true = true ']' 2024-09-18T15:24:59.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-09-18T15:24:59.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:24:59.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:24:59.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-09-18T15:24:59.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:24:59.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/8 2024-09-18T15:24:59.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:24:59.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:24:59.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:24:59.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:24:59.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-09-18T15:24:59.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-09-18T15:24:59.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:24:59.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:24:59.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:24:59.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:24:59.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:24:59.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:24:59.154 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:24:59.154 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:24:59.154 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:24:59.156 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:24:59.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:24:59.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:24:59.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:24:59.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:24:59.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:24:59.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:24:59.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:24:59.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:24:59.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:24:59.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:24:59.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:24:59.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:24:59.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/8 2024-09-18T15:24:59.157 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:24:59.158 INFO:tasks.workunit.client.0.smithi046.stdout:add osd8 088a6f10-7ff3-4fd7-8661-73467b20133e 2024-09-18T15:24:59.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=088a6f10-7ff3-4fd7-8661-73467b20133e 2024-09-18T15:24:59.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 088a6f10-7ff3-4fd7-8661-73467b20133e' 2024-09-18T15:24:59.158 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:24:59.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDL8OpmlRwSChAA0ayq3pc3n5Q4xwJEYsLpzQ== 2024-09-18T15:24:59.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDL8OpmlRwSChAA0ayq3pc3n5Q4xwJEYsLpzQ=="}' 2024-09-18T15:24:59.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 088a6f10-7ff3-4fd7-8661-73467b20133e -i td/osd-scrub-repair/8/new.json 2024-09-18T15:24:59.658 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:24:59.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/8/new.json 2024-09-18T15:24:59.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDL8OpmlRwSChAA0ayq3pc3n5Q4xwJEYsLpzQ== --osd-uuid 088a6f10-7ff3-4fd7-8661-73467b20133e 2024-09-18T15:24:59.711 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:24:59.703+0000 7fb4c1fd7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:24:59.714 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:24:59.707+0000 7fb4c1fd7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:24:59.716 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:24:59.707+0000 7fb4c1fd7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:24:59.717 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:24:59.711+0000 7fb4c1fd7540 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-09-18T15:24:59.718 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:24:59.711+0000 7fb4c1fd7540 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-09-18T15:24:59.718 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:24:59.711+0000 7fb4c1fd7540 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-09-18T15:24:59.792 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:24:59.783+0000 7fb4c1fd7540 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-09-18T15:25:06.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-09-18T15:25:06.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:25:06.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-09-18T15:25:06.469 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd8 key to auth repository 2024-09-18T15:25:06.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:25:06.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-09-18T15:25:06.854 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.8 2024-09-18T15:25:06.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:25:06.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:25:06.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:25:06.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:25:06.879 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:25:06.870+0000 7fe918d3d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:25:06.882 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:25:06.874+0000 7fe918d3d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:25:06.885 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:25:06.874+0000 7fe918d3d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:25:07.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-09-18T15:25:07.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:25:07.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-09-18T15:25:07.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:25:07.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:25:07.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:07.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:25:07.249 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:25:07.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:07.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T15:25:07.436 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:25:07.426+0000 7fe918d3d540 -1 Falling back to public interface 2024-09-18T15:25:07.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:08.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:08.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:08.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:25:08.629 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:25:08.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:08.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T15:25:08.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:09.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:09.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:09.964 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:25:09.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:25:09.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:09.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T15:25:10.237 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:25:10.226+0000 7fe918d3d540 -1 osd.8 0 log_to_monitors true 2024-09-18T15:25:10.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:11.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:11.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:11.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:25:11.287 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:25:11.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:11.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T15:25:11.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:12.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:12.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:12.703 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:25:12.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:25:12.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:12.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T15:25:13.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:14.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:14.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:14.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:25:14.248 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:25:14.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:14.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T15:25:14.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:15.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:15.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:15.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:25:15.645 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:25:15.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:15.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T15:25:16.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:17.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:17.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:17.108 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:25:17.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:25:17.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:17.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T15:25:17.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:18.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:18.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:18.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:25:18.435 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:25:18.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:18.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T15:25:18.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:19.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:19.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:19.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:25:19.745 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:25:19.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:19.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T15:25:20.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:21.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:21.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:21.062 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:25:21.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:25:21.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:21.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T15:25:21.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:22.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:22.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:22.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:25:22.382 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:25:22.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:22.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T15:25:22.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:23.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:23.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:23.697 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:25:23.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:25:23.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:23.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T15:25:24.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:25.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:25.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:25.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:25:25.415 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:25:25.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:25.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T15:25:26.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:27.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:27.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:27.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:25:27.112 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:25:27.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:27.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T15:25:28.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:29.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:29.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:29.047 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:25:29.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:25:29.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:29.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T15:25:29.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:30.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:30.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:30.615 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:25:30.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:25:30.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T15:25:30.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:31.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:32.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:32.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:32.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:25:32.403 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:25:32.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:32.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T15:25:33.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:34.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:34.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:34.025 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:25:34.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:25:34.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:34.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T15:25:34.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:35.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:35.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:35.346 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:25:35.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:25:35.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:35.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T15:25:35.780 INFO:tasks.workunit.client.0.smithi046.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/288247,v1:127.0.0.1:6867/288247] [v2:127.0.0.1:6868/288247,v1:127.0.0.1:6869/288247] exists,up 088a6f10-7ff3-4fd7-8661-73467b20133e 2024-09-18T15:25:35.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:25:35.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:25:35.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:25:35.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-18T15:25:35.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: '[' true = true ']' 2024-09-18T15:25:35.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-09-18T15:25:35.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:25:35.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:25:35.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-09-18T15:25:35.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:25:35.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/9 2024-09-18T15:25:35.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:25:35.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:25:35.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:25:35.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:25:35.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-18T15:25:35.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-18T15:25:35.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:25:35.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:25:35.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:25:35.784 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:25:35.784 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:25:35.784 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:25:35.786 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:25:35.786 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:25:35.786 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:25:35.786 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:25:35.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:25:35.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:25:35.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:25:35.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:25:35.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:25:35.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:25:35.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:25:35.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:25:35.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:25:35.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:25:35.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:25:35.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:25:35.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/9 2024-09-18T15:25:35.788 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:25:35.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=364a066e-2eb8-45f2-a30f-be7da6149b02 2024-09-18T15:25:35.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 364a066e-2eb8-45f2-a30f-be7da6149b02' 2024-09-18T15:25:35.790 INFO:tasks.workunit.client.0.smithi046.stdout:add osd9 364a066e-2eb8-45f2-a30f-be7da6149b02 2024-09-18T15:25:35.791 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:25:35.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDv8OpmqurmLxAAEyXt5Ntt3g62NK3bGQ0a7w== 2024-09-18T15:25:35.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDv8OpmqurmLxAAEyXt5Ntt3g62NK3bGQ0a7w=="}' 2024-09-18T15:25:35.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 364a066e-2eb8-45f2-a30f-be7da6149b02 -i td/osd-scrub-repair/9/new.json 2024-09-18T15:25:36.324 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:25:36.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/9/new.json 2024-09-18T15:25:36.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDv8OpmqurmLxAAEyXt5Ntt3g62NK3bGQ0a7w== --osd-uuid 364a066e-2eb8-45f2-a30f-be7da6149b02 2024-09-18T15:25:36.379 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:25:36.370+0000 7f301e63d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:25:36.436 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:25:36.426+0000 7f301e63d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:25:36.438 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:25:36.430+0000 7f301e63d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:25:36.439 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:25:36.430+0000 7f301e63d540 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-09-18T15:25:36.439 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:25:36.430+0000 7f301e63d540 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-09-18T15:25:36.439 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:25:36.430+0000 7f301e63d540 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-09-18T15:25:36.520 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:25:36.510+0000 7f301e63d540 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-09-18T15:25:43.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-09-18T15:25:43.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:25:43.793 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd9 key to auth repository 2024-09-18T15:25:43.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-09-18T15:25:43.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:25:44.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-09-18T15:25:44.184 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.9 2024-09-18T15:25:44.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:25:44.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:25:44.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:25:44.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:25:44.207 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:25:44.194+0000 7f33c9f83540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:25:44.208 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:25:44.198+0000 7f33c9f83540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:25:44.209 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:25:44.198+0000 7f33c9f83540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:25:44.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-09-18T15:25:44.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:25:44.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-09-18T15:25:44.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:25:44.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:25:44.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:44.531 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:25:44.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:25:44.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:44.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:25:44.751 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:25:44.742+0000 7f33c9f83540 -1 Falling back to public interface 2024-09-18T15:25:44.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:45.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:45.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:45.831 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:25:45.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:25:45.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:45.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:25:46.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:47.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:47.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:47.143 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:25:47.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:25:47.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:47.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:25:47.390 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:25:47.382+0000 7f33c9f83540 -1 osd.9 0 log_to_monitors true 2024-09-18T15:25:47.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:48.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:48.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:48.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:25:48.455 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:25:48.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:48.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:25:48.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:49.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:49.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:49.777 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:25:49.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:25:49.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:25:49.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:50.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:51.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:51.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:51.163 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:25:51.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:25:51.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:25:51.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:51.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:52.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:52.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:52.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:25:52.491 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:25:52.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:52.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:25:52.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:53.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:53.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:53.814 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:25:53.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:25:53.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:53.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:25:54.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:55.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:55.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:55.133 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:25:55.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:25:55.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:55.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:25:55.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:56.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:56.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:56.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:25:56.484 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:25:56.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:56.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:25:56.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:57.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:57.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:57.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:25:57.809 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:25:57.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:57.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:25:58.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:59.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:59.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:59.129 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:25:59.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:25:59.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:59.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:25:59.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:00.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:00.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:00.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:26:00.527 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:26:00.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:00.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:26:00.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:01.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:01.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:01.844 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:26:01.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:26:01.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:01.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:26:02.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:03.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:03.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:03.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:26:03.496 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:26:03.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:03.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:26:04.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:05.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:05.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:05.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:26:05.231 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:26:05.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:05.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:26:05.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:06.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:06.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:06.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:26:06.775 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:26:06.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:06.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:26:07.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:08.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:08.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:08.285 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:26:08.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:26:08.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:08.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:26:08.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:09.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:09.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:09.910 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:26:09.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:26:09.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:09.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:26:10.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:11.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:11.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:11.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:26:11.570 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:26:11.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:11.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:26:11.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:12.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:12.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:12.970 INFO:tasks.workunit.client.0.smithi046.stdout:20 2024-09-18T15:26:12.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-18T15:26:12.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:12.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:26:13.295 INFO:tasks.workunit.client.0.smithi046.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/290051,v1:127.0.0.1:6875/290051] [v2:127.0.0.1:6876/290051,v1:127.0.0.1:6877/290051] exists,up 364a066e-2eb8-45f2-a30f-be7da6149b02 2024-09-18T15:26:13.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:26:13.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:26:13.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:26:13.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:843: corrupt_and_repair_lrc: create_rbd_pool 2024-09-18T15:26:13.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T15:26:13.716 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' does not exist 2024-09-18T15:26:13.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T15:26:13.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T15:26:14.182 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' created 2024-09-18T15:26:14.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:26:15.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T15:26:18.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:844: corrupt_and_repair_lrc: wait_for_clean 2024-09-18T15:26:18.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:26:18.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:26:18.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:26:18.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:26:18.974 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:26:18.975 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:26:18.975 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:26:18.975 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:26:18.975 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:26:18.975 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:26:19.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:26:19.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:26:19.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:26:19.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:26:19.197 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:26:19.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:26:19.502 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:26:19.502 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:26:19.502 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:26:19.502 INFO:tasks.workunit.client.0.smithi046.stderr:4 2024-09-18T15:26:19.502 INFO:tasks.workunit.client.0.smithi046.stderr:5 2024-09-18T15:26:19.502 INFO:tasks.workunit.client.0.smithi046.stderr:6 2024-09-18T15:26:19.503 INFO:tasks.workunit.client.0.smithi046.stderr:7 2024-09-18T15:26:19.503 INFO:tasks.workunit.client.0.smithi046.stderr:8 2024-09-18T15:26:19.503 INFO:tasks.workunit.client.0.smithi046.stderr:9' 2024-09-18T15:26:19.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:26:19.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:19.503 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:26:19.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836556 2024-09-18T15:26:19.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836556 2024-09-18T15:26:19.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836556' 2024-09-18T15:26:19.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:19.645 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:26:19.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673026 2024-09-18T15:26:19.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673026 2024-09-18T15:26:19.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836556 1-42949673026' 2024-09-18T15:26:19.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:19.864 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:26:20.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509502 2024-09-18T15:26:20.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509502 2024-09-18T15:26:20.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836556 1-42949673026 2-64424509502' 2024-09-18T15:26:20.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:20.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:26:20.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345978 2024-09-18T15:26:20.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345978 2024-09-18T15:26:20.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836556 1-42949673026 2-64424509502 3-85899345978' 2024-09-18T15:26:20.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:20.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:26:20.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182447 2024-09-18T15:26:20.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182447 2024-09-18T15:26:20.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836556 1-42949673026 2-64424509502 3-85899345978 4-107374182447' 2024-09-18T15:26:20.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:20.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:26:20.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018923 2024-09-18T15:26:20.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018923 2024-09-18T15:26:20.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836556 1-42949673026 2-64424509502 3-85899345978 4-107374182447 5-128849018923' 2024-09-18T15:26:20.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:20.454 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:26:20.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855394 2024-09-18T15:26:20.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855394 2024-09-18T15:26:20.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836556 1-42949673026 2-64424509502 3-85899345978 4-107374182447 5-128849018923 6-150323855394' 2024-09-18T15:26:20.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:20.595 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:26:20.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691866 2024-09-18T15:26:20.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691866 2024-09-18T15:26:20.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836556 1-42949673026 2-64424509502 3-85899345978 4-107374182447 5-128849018923 6-150323855394 7-171798691866' 2024-09-18T15:26:20.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:20.732 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:26:20.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528338 2024-09-18T15:26:20.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528338 2024-09-18T15:26:20.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836556 1-42949673026 2-64424509502 3-85899345978 4-107374182447 5-128849018923 6-150323855394 7-171798691866 8-193273528338' 2024-09-18T15:26:20.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:20.874 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:26:21.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=214748364805 2024-09-18T15:26:21.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 214748364805 2024-09-18T15:26:21.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836556 1-42949673026 2-64424509502 3-85899345978 4-107374182447 5-128849018923 6-150323855394 7-171798691866 8-193273528338 9-214748364805' 2024-09-18T15:26:21.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:21.016 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836556 2024-09-18T15:26:21.016 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:21.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:26:21.019 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836556 2024-09-18T15:26:21.019 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:21.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836556 2024-09-18T15:26:21.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836556' 2024-09-18T15:26:21.022 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836556 2024-09-18T15:26:21.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:26:21.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836556 -lt 21474836556 2024-09-18T15:26:21.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:21.327 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673026 2024-09-18T15:26:21.327 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:21.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:26:21.331 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673026 2024-09-18T15:26:21.331 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:21.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673026 2024-09-18T15:26:21.333 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949673026 2024-09-18T15:26:21.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673026' 2024-09-18T15:26:21.334 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:26:21.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673026 -lt 42949673026 2024-09-18T15:26:21.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:21.736 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509502 2024-09-18T15:26:21.736 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:21.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:26:21.740 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509502 2024-09-18T15:26:21.740 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:21.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509502 2024-09-18T15:26:21.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509502' 2024-09-18T15:26:21.742 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509502 2024-09-18T15:26:21.742 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:26:22.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509503 -lt 64424509502 2024-09-18T15:26:22.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:22.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345978 2024-09-18T15:26:22.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:22.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:26:22.084 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345978 2024-09-18T15:26:22.084 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:22.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345978 2024-09-18T15:26:22.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345978' 2024-09-18T15:26:22.086 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.3 seq 85899345978 2024-09-18T15:26:22.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:26:22.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345978 -lt 85899345978 2024-09-18T15:26:22.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:22.488 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182447 2024-09-18T15:26:22.488 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:22.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:26:22.492 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182447 2024-09-18T15:26:22.492 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:22.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182447 2024-09-18T15:26:22.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182447' 2024-09-18T15:26:22.494 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.4 seq 107374182447 2024-09-18T15:26:22.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:26:22.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182447 -lt 107374182447 2024-09-18T15:26:22.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:22.848 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018923 2024-09-18T15:26:22.848 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:22.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:26:22.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018923 2024-09-18T15:26:22.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:22.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018923 2024-09-18T15:26:22.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018923' 2024-09-18T15:26:22.854 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.5 seq 128849018923 2024-09-18T15:26:22.854 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:26:23.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018923 -lt 128849018923 2024-09-18T15:26:23.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:23.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855394 2024-09-18T15:26:23.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:23.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:26:23.158 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855394 2024-09-18T15:26:23.158 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:23.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855394 2024-09-18T15:26:23.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855394' 2024-09-18T15:26:23.160 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.6 seq 150323855394 2024-09-18T15:26:23.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:26:23.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855395 -lt 150323855394 2024-09-18T15:26:23.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:23.494 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691866 2024-09-18T15:26:23.494 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:23.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:26:23.498 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691866 2024-09-18T15:26:23.499 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:23.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691866 2024-09-18T15:26:23.500 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.7 seq 171798691866 2024-09-18T15:26:23.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691866' 2024-09-18T15:26:23.501 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:26:23.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691866 -lt 171798691866 2024-09-18T15:26:23.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:23.861 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528338 2024-09-18T15:26:23.861 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:23.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:26:23.864 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528338 2024-09-18T15:26:23.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:23.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528338 2024-09-18T15:26:23.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528338' 2024-09-18T15:26:23.866 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.8 seq 193273528338 2024-09-18T15:26:23.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:26:24.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528338 -lt 193273528338 2024-09-18T15:26:24.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:24.185 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-214748364805 2024-09-18T15:26:24.186 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:24.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:26:24.189 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-214748364805 2024-09-18T15:26:24.190 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:24.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=214748364805 2024-09-18T15:26:24.192 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.9 seq 214748364805 2024-09-18T15:26:24.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 214748364805' 2024-09-18T15:26:24.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:26:24.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 214748364805 -lt 214748364805 2024-09-18T15:26:24.552 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:26:24.553 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:26:24.553 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:26:24.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T15:26:24.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:26:24.977 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:26:24.977 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:26:24.977 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:26:24.977 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:26:24.977 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:26:24.977 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:26:25.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T15:26:25.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:26:25.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:26:25.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:26:25.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T15:26:25.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:26:25.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:26:25.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: corrupt_and_repair_lrc: create_ec_pool ecpool true k=4 m=2 l=3 plugin=lrc 2024-09-18T15:26:25.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: create_ec_pool: local pool_name=ecpool 2024-09-18T15:26:25.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: create_ec_pool: shift 2024-09-18T15:26:25.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: create_ec_pool: local allow_overwrites=true 2024-09-18T15:26:25.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: create_ec_pool: shift 2024-09-18T15:26:25.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-09-18T15:26:26.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-18T15:26:26.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-18T15:26:27.537 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'ecpool' created 2024-09-18T15:26:27.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:26:28.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: create_ec_pool: '[' true = true ']' 2024-09-18T15:26:28.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-18T15:26:29.209 INFO:tasks.workunit.client.0.smithi046.stderr:set pool 2 allow_ec_overwrites to true 2024-09-18T15:26:29.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: create_ec_pool: wait_for_clean 2024-09-18T15:26:29.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:26:29.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:26:29.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:26:29.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:26:29.223 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:26:29.224 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:26:29.224 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:26:29.225 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:26:29.225 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:26:29.225 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:26:29.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:26:29.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:26:29.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:26:29.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:26:29.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:26:29.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:26:29.908 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:26:29.908 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:26:29.908 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:26:29.908 INFO:tasks.workunit.client.0.smithi046.stderr:4 2024-09-18T15:26:29.908 INFO:tasks.workunit.client.0.smithi046.stderr:5 2024-09-18T15:26:29.908 INFO:tasks.workunit.client.0.smithi046.stderr:6 2024-09-18T15:26:29.908 INFO:tasks.workunit.client.0.smithi046.stderr:7 2024-09-18T15:26:29.908 INFO:tasks.workunit.client.0.smithi046.stderr:8 2024-09-18T15:26:29.909 INFO:tasks.workunit.client.0.smithi046.stderr:9' 2024-09-18T15:26:29.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:26:29.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:29.909 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:26:30.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836559 2024-09-18T15:26:30.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836559 2024-09-18T15:26:30.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836559' 2024-09-18T15:26:30.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:30.052 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:26:30.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673029 2024-09-18T15:26:30.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673029 2024-09-18T15:26:30.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836559 1-42949673029' 2024-09-18T15:26:30.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:30.184 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:26:30.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509506 2024-09-18T15:26:30.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509506 2024-09-18T15:26:30.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836559 1-42949673029 2-64424509506' 2024-09-18T15:26:30.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:30.321 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:26:30.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345981 2024-09-18T15:26:30.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345981 2024-09-18T15:26:30.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836559 1-42949673029 2-64424509506 3-85899345981' 2024-09-18T15:26:30.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:30.463 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:26:30.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182450 2024-09-18T15:26:30.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182450 2024-09-18T15:26:30.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836559 1-42949673029 2-64424509506 3-85899345981 4-107374182450' 2024-09-18T15:26:30.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:30.606 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:26:30.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018926 2024-09-18T15:26:30.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018926 2024-09-18T15:26:30.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836559 1-42949673029 2-64424509506 3-85899345981 4-107374182450 5-128849018926' 2024-09-18T15:26:30.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:30.748 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:26:30.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855397 2024-09-18T15:26:30.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855397 2024-09-18T15:26:30.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836559 1-42949673029 2-64424509506 3-85899345981 4-107374182450 5-128849018926 6-150323855397' 2024-09-18T15:26:30.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:30.892 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:26:31.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691869 2024-09-18T15:26:31.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691869 2024-09-18T15:26:31.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836559 1-42949673029 2-64424509506 3-85899345981 4-107374182450 5-128849018926 6-150323855397 7-171798691869' 2024-09-18T15:26:31.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:31.056 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:26:31.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528341 2024-09-18T15:26:31.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528341 2024-09-18T15:26:31.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836559 1-42949673029 2-64424509506 3-85899345981 4-107374182450 5-128849018926 6-150323855397 7-171798691869 8-193273528341' 2024-09-18T15:26:31.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:31.201 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:26:31.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=214748364808 2024-09-18T15:26:31.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 214748364808 2024-09-18T15:26:31.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836559 1-42949673029 2-64424509506 3-85899345981 4-107374182450 5-128849018926 6-150323855397 7-171798691869 8-193273528341 9-214748364808' 2024-09-18T15:26:31.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:31.343 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836559 2024-09-18T15:26:31.344 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:31.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:26:31.347 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836559 2024-09-18T15:26:31.347 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:31.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836559 2024-09-18T15:26:31.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836559' 2024-09-18T15:26:31.349 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836559 2024-09-18T15:26:31.350 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:26:31.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836559 -lt 21474836559 2024-09-18T15:26:31.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:31.679 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673029 2024-09-18T15:26:31.680 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:31.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:26:31.683 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673029 2024-09-18T15:26:31.684 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:31.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673029 2024-09-18T15:26:31.686 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949673029 2024-09-18T15:26:31.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673029' 2024-09-18T15:26:31.687 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:26:32.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673029 -lt 42949673029 2024-09-18T15:26:32.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:32.039 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509506 2024-09-18T15:26:32.040 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:32.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:26:32.042 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509506 2024-09-18T15:26:32.043 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:32.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509506 2024-09-18T15:26:32.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509506' 2024-09-18T15:26:32.045 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509506 2024-09-18T15:26:32.046 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:26:32.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509506 -lt 64424509506 2024-09-18T15:26:32.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:32.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345981 2024-09-18T15:26:32.406 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:32.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:26:32.409 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345981 2024-09-18T15:26:32.409 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:32.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345981 2024-09-18T15:26:32.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345981' 2024-09-18T15:26:32.412 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.3 seq 85899345981 2024-09-18T15:26:32.412 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:26:32.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345981 -lt 85899345981 2024-09-18T15:26:32.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:32.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182450 2024-09-18T15:26:32.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:32.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:26:32.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182450 2024-09-18T15:26:32.712 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:32.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182450 2024-09-18T15:26:32.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182450' 2024-09-18T15:26:32.714 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.4 seq 107374182450 2024-09-18T15:26:32.715 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:26:33.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182450 -lt 107374182450 2024-09-18T15:26:33.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:33.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018926 2024-09-18T15:26:33.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:33.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:26:33.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:33.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018926 2024-09-18T15:26:33.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018926 2024-09-18T15:26:33.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018926' 2024-09-18T15:26:33.018 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.5 seq 128849018926 2024-09-18T15:26:33.019 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:26:33.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018926 -lt 128849018926 2024-09-18T15:26:33.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:33.335 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855397 2024-09-18T15:26:33.335 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:33.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:26:33.338 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855397 2024-09-18T15:26:33.338 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:33.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855397 2024-09-18T15:26:33.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855397' 2024-09-18T15:26:33.341 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.6 seq 150323855397 2024-09-18T15:26:33.341 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:26:33.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855398 -lt 150323855397 2024-09-18T15:26:33.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:33.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691869 2024-09-18T15:26:33.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:33.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:26:33.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:33.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691869 2024-09-18T15:26:33.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691869 2024-09-18T15:26:33.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691869' 2024-09-18T15:26:33.698 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.7 seq 171798691869 2024-09-18T15:26:33.699 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:26:34.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691869 -lt 171798691869 2024-09-18T15:26:34.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:34.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528341 2024-09-18T15:26:34.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:34.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:26:34.020 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528341 2024-09-18T15:26:34.021 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:34.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528341 2024-09-18T15:26:34.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528341' 2024-09-18T15:26:34.023 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.8 seq 193273528341 2024-09-18T15:26:34.024 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:26:34.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528341 -lt 193273528341 2024-09-18T15:26:34.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:34.391 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-214748364808 2024-09-18T15:26:34.391 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:34.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:26:34.394 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-214748364808 2024-09-18T15:26:34.394 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:34.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=214748364808 2024-09-18T15:26:34.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 214748364808' 2024-09-18T15:26:34.397 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.9 seq 214748364808 2024-09-18T15:26:34.397 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:26:34.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 214748364808 -lt 214748364808 2024-09-18T15:26:34.758 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:26:34.758 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:26:34.759 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:26:35.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:26:35.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:26:35.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:26:35.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:26:35.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:26:35.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:26:35.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:26:35.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:26:35.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:26:35.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:26:35.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:26:35.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:26:35.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:26:35.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:26:35.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:26:35.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: create_ec_pool: return 0 2024-09-18T15:26:35.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-09-18T15:26:35.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-18T15:26:35.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-09-18T15:26:35.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:250: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-18T15:26:35.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:26:35.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2024-09-18T15:26:35.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=SOMETHING 2024-09-18T15:26:35.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:26:35.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:26:35.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:26:36.395 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:26:36.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:26:37.003 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:26:37.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:26:37.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:26:37.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-18T15:26:37.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:252: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-09-18T15:26:37.362 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-09-18T15:26:37.362 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T15:26:37.362 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:26:37.362 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:26:37.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:252: corrupt_and_repair_erasure_coded: local primary=3 2024-09-18T15:26:37.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: osds=($(get_osds $poolname SOMETHING | sed -e "s/$primary//")) 2024-09-18T15:26:37.676 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-09-18T15:26:37.676 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-09-18T15:26:37.676 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-18T15:26:37.677 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-09-18T15:26:37.677 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:26:37.678 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:26:38.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:26:38.031 INFO:tasks.workunit.client.0.smithi046.stderr:5 2024-09-18T15:26:38.031 INFO:tasks.workunit.client.0.smithi046.stderr:9 2024-09-18T15:26:38.031 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:26:38.031 INFO:tasks.workunit.client.0.smithi046.stderr:6 2024-09-18T15:26:38.031 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:26:38.031 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:26:38.031 INFO:tasks.workunit.client.0.smithi046.stderr:7' 2024-09-18T15:26:38.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 6 2 1 7 2024-09-18T15:26:38.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local -a osds 2024-09-18T15:26:38.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-09-18T15:26:38.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-09-18T15:26:38.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-09-18T15:26:38.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:225: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-18T15:26:38.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local poolname=ecpool 2024-09-18T15:26:38.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local osd=3 2024-09-18T15:26:38.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:26:38.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:26:38.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:26:38.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-18T15:26:38.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:26:38.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:26:38.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:26:38.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:26:38.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-18T15:26:38.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:26:38.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-18T15:26:38.034 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:26:38.034 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:26:38.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:26:38.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:26:38.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:26:38.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:26:38.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:26:38.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:26:38.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:26:38.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-18T15:26:38.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:26:38.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:26:38.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-18T15:26:40.619 INFO:tasks.workunit.client.0.smithi046.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-09-18T15:26:41.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-18T15:26:41.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:26:41.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:26:41.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T15:26:41.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:26:41.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:26:41.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:26:41.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:26:41.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:26:41.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:26:41.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-18T15:26:41.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-18T15:26:41.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:26:41.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:26:41.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:26:41.162 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:26:41.162 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:26:41.162 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:26:41.162 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:26:41.162 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:26:41.163 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:26:41.163 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:26:41.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:26:41.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:26:41.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:26:41.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:26:41.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:26:41.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:26:41.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:26:41.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:26:41.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:26:41.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:26:41.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-18T15:26:41.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T15:26:41.167 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.3 2024-09-18T15:26:41.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:26:41.168 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-18T15:26:41.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T15:26:41.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:26:41.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:26:41.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:26:41.198 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:26:41.189+0000 7fef104f2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:26:41.202 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:26:41.193+0000 7fef104f2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:26:41.205 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:26:41.193+0000 7fef104f2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:26:41.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T15:26:41.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:26:41.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:26:41.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:26:41.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:26:41.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:41.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:26:41.473 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:26:41.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:41.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:26:41.740 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:26:41.729+0000 7fef104f2540 -1 Falling back to public interface 2024-09-18T15:26:41.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:42.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:42.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:42.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:26:42.867 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:26:42.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:42.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:26:43.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:44.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:44.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:44.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:26:44.207 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:26:44.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:44.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:26:44.362 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:26:44.353+0000 7fef104f2540 -1 osd.3 68 log_to_monitors true 2024-09-18T15:26:44.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:45.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:45.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:45.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:26:45.546 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:26:45.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:45.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:26:45.885 INFO:tasks.workunit.client.0.smithi046.stderr:osd.3 up in weight 1 up_from 72 up_thru 72 down_at 69 last_clean_interval [20,68) [v2:127.0.0.1:6826/293149,v1:127.0.0.1:6827/293149] [v2:127.0.0.1:6828/293149,v1:127.0.0.1:6829/293149] exists,up 105fcb65-1d2b-4358-9b52-7d1b65a95f69 2024-09-18T15:26:45.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:26:45.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:26:45.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:26:45.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:26:45.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:26:45.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:26:45.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:26:45.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:26:45.887 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:26:45.887 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:26:45.887 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:26:45.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:26:45.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:26:45.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:26:46.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:26:46.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:26:46.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:26:46.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:26:46.151 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:26:46.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:26:46.478 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:26:46.478 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:26:46.478 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:26:46.478 INFO:tasks.workunit.client.0.smithi046.stderr:4 2024-09-18T15:26:46.478 INFO:tasks.workunit.client.0.smithi046.stderr:5 2024-09-18T15:26:46.478 INFO:tasks.workunit.client.0.smithi046.stderr:6 2024-09-18T15:26:46.478 INFO:tasks.workunit.client.0.smithi046.stderr:7 2024-09-18T15:26:46.478 INFO:tasks.workunit.client.0.smithi046.stderr:8 2024-09-18T15:26:46.478 INFO:tasks.workunit.client.0.smithi046.stderr:9' 2024-09-18T15:26:46.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:26:46.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:46.479 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:26:46.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836563 2024-09-18T15:26:46.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836563 2024-09-18T15:26:46.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836563' 2024-09-18T15:26:46.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:46.670 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:26:46.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673033 2024-09-18T15:26:46.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673033 2024-09-18T15:26:46.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836563 1-42949673033' 2024-09-18T15:26:46.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:46.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:26:47.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509510 2024-09-18T15:26:47.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509510 2024-09-18T15:26:47.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836563 1-42949673033 2-64424509510' 2024-09-18T15:26:47.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:47.007 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:26:47.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=309237645314 2024-09-18T15:26:47.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 309237645314 2024-09-18T15:26:47.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836563 1-42949673033 2-64424509510 3-309237645314' 2024-09-18T15:26:47.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:47.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:26:47.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182454 2024-09-18T15:26:47.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182454 2024-09-18T15:26:47.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836563 1-42949673033 2-64424509510 3-309237645314 4-107374182454' 2024-09-18T15:26:47.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:47.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:26:47.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018930 2024-09-18T15:26:47.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018930 2024-09-18T15:26:47.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836563 1-42949673033 2-64424509510 3-309237645314 4-107374182454 5-128849018930' 2024-09-18T15:26:47.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:47.422 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:26:47.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855402 2024-09-18T15:26:47.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855402 2024-09-18T15:26:47.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836563 1-42949673033 2-64424509510 3-309237645314 4-107374182454 5-128849018930 6-150323855402' 2024-09-18T15:26:47.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:47.564 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:26:47.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691873 2024-09-18T15:26:47.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691873 2024-09-18T15:26:47.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836563 1-42949673033 2-64424509510 3-309237645314 4-107374182454 5-128849018930 6-150323855402 7-171798691873' 2024-09-18T15:26:47.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:47.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:26:47.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528345 2024-09-18T15:26:47.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528345 2024-09-18T15:26:47.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836563 1-42949673033 2-64424509510 3-309237645314 4-107374182454 5-128849018930 6-150323855402 7-171798691873 8-193273528345' 2024-09-18T15:26:47.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:26:47.883 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:26:48.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=214748364813 2024-09-18T15:26:48.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 214748364813 2024-09-18T15:26:48.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836563 1-42949673033 2-64424509510 3-309237645314 4-107374182454 5-128849018930 6-150323855402 7-171798691873 8-193273528345 9-214748364813' 2024-09-18T15:26:48.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:48.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836563 2024-09-18T15:26:48.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:48.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:26:48.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836563 2024-09-18T15:26:48.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:48.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836563 2024-09-18T15:26:48.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836563' 2024-09-18T15:26:48.028 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836563 2024-09-18T15:26:48.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:26:48.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836563 -lt 21474836563 2024-09-18T15:26:48.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:48.341 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673033 2024-09-18T15:26:48.341 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:48.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:26:48.343 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673033 2024-09-18T15:26:48.344 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:48.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673033 2024-09-18T15:26:48.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673033' 2024-09-18T15:26:48.346 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 42949673033 2024-09-18T15:26:48.347 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:26:48.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673033 -lt 42949673033 2024-09-18T15:26:48.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:48.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509510 2024-09-18T15:26:48.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:48.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:26:48.659 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509510 2024-09-18T15:26:48.659 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:48.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509510 2024-09-18T15:26:48.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509510' 2024-09-18T15:26:48.661 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509510 2024-09-18T15:26:48.662 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:26:49.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509509 -lt 64424509510 2024-09-18T15:26:49.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:26:50.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:26:50.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:26:50.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509510 -lt 64424509510 2024-09-18T15:26:50.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:50.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-309237645314 2024-09-18T15:26:50.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:50.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:26:50.328 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-309237645314 2024-09-18T15:26:50.329 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:50.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=309237645314 2024-09-18T15:26:50.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 309237645314' 2024-09-18T15:26:50.331 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.3 seq 309237645314 2024-09-18T15:26:50.332 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:26:50.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 309237645314 -lt 309237645314 2024-09-18T15:26:50.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:50.689 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182454 2024-09-18T15:26:50.689 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:50.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:26:50.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182454 2024-09-18T15:26:50.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:50.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182454 2024-09-18T15:26:50.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182454' 2024-09-18T15:26:50.695 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.4 seq 107374182454 2024-09-18T15:26:50.696 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:26:51.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182455 -lt 107374182454 2024-09-18T15:26:51.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:51.041 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018930 2024-09-18T15:26:51.042 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:51.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:26:51.045 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018930 2024-09-18T15:26:51.046 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:51.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018930 2024-09-18T15:26:51.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018930' 2024-09-18T15:26:51.049 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.5 seq 128849018930 2024-09-18T15:26:51.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:26:51.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018931 -lt 128849018930 2024-09-18T15:26:51.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:51.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855402 2024-09-18T15:26:51.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:51.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:26:51.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855402 2024-09-18T15:26:51.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:51.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855402 2024-09-18T15:26:51.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855402' 2024-09-18T15:26:51.366 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.6 seq 150323855402 2024-09-18T15:26:51.367 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:26:51.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855402 -lt 150323855402 2024-09-18T15:26:51.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:51.735 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691873 2024-09-18T15:26:51.735 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:51.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:26:51.738 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691873 2024-09-18T15:26:51.738 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:51.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691873 2024-09-18T15:26:51.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691873' 2024-09-18T15:26:51.740 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.7 seq 171798691873 2024-09-18T15:26:51.740 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:26:52.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691874 -lt 171798691873 2024-09-18T15:26:52.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:52.078 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528345 2024-09-18T15:26:52.078 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:52.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:26:52.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528345 2024-09-18T15:26:52.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:52.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528345 2024-09-18T15:26:52.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528345' 2024-09-18T15:26:52.084 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.8 seq 193273528345 2024-09-18T15:26:52.084 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:26:52.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528346 -lt 193273528345 2024-09-18T15:26:52.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:26:52.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-214748364813 2024-09-18T15:26:52.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:26:52.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:26:52.438 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-214748364813 2024-09-18T15:26:52.438 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:26:52.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=214748364813 2024-09-18T15:26:52.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 214748364813' 2024-09-18T15:26:52.440 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.9 seq 214748364813 2024-09-18T15:26:52.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:26:52.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 214748364813 -lt 214748364813 2024-09-18T15:26:52.791 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:26:52.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:26:52.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:26:53.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:26:53.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:26:53.163 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:26:53.163 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:26:53.163 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:26:53.163 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:26:53.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:26:53.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:26:53.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:26:53.558 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:26:53.559 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:26:53.559 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:26:53.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:26:53.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:26:53.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:26:53.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-18T15:26:53.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-09-18T15:26:53.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T15:26:53.983 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:26:53.983 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:26:54.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: local pg=2.0 2024-09-18T15:26:54.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: repair 2.0 2024-09-18T15:26:54.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-09-18T15:26:54.341 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-09-18T15:26:54.341 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:26:54.342 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:26:54.342 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:26:54.342 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:26:54.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T15:26:27.290508+0000 2024-09-18T15:26:54.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-09-18T15:26:54.950 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-18T15:26:54.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-09-18T15:26:27.290508+0000 2024-09-18T15:26:54.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=2.0 2024-09-18T15:26:54.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:26:27.290508+0000 2024-09-18T15:26:54.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:26:54.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:26:54.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:26:54.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:26:54.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:26:54.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:26:54.965 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:26:54.965 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:26:55.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:26:27.290508+0000 '>' 2024-09-18T15:26:27.290508+0000 2024-09-18T15:26:55.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:26:56.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:26:56.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:26:56.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:26:56.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:26:56.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:26:56.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:26:56.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:26:56.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:26:27.290508+0000 '>' 2024-09-18T15:26:27.290508+0000 2024-09-18T15:26:56.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:26:57.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:26:57.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:26:57.592 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:26:57.592 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:26:57.592 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:26:57.592 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:26:57.592 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:26:57.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:26:27.290508+0000 '>' 2024-09-18T15:26:27.290508+0000 2024-09-18T15:26:57.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:26:58.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:26:58.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:26:58.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:26:58.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:26:58.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:26:58.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:26:58.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:26:59.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:26:27.290508+0000 '>' 2024-09-18T15:26:27.290508+0000 2024-09-18T15:26:59.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:27:00.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:27:00.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:27:00.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:27:00.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:27:00.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:27:00.221 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:27:00.221 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:27:00.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:26:27.290508+0000 '>' 2024-09-18T15:26:27.290508+0000 2024-09-18T15:27:00.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:27:01.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:27:01.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:27:01.536 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:27:01.536 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:27:01.536 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:27:01.537 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:27:01.537 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:27:01.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:26:55.382803+0000 '>' 2024-09-18T15:26:27.290508+0000 2024-09-18T15:27:01.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:27:01.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:241: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-18T15:27:01.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:27:01.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:27:01.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-09-18T15:27:01.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:27:01.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-18T15:27:01.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:27:01.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:27:01.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-09-18T15:27:01.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:27:01.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-18T15:27:01.855 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:27:01.855 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:27:01.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:27:01.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:27:01.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:27:02.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:27:02.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-18T15:27:02.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:27:02.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:27:02.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-09-18T15:27:02.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:27:02.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-18T15:27:02.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-18T15:27:04.561 INFO:tasks.workunit.client.0.smithi046.stdout:_ 2024-09-18T15:27:04.561 INFO:tasks.workunit.client.0.smithi046.stdout:hinfo_key 2024-09-18T15:27:04.561 INFO:tasks.workunit.client.0.smithi046.stdout:snapset 2024-09-18T15:27:05.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-18T15:27:05.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:27:05.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:27:05.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-09-18T15:27:05.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:27:05.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-18T15:27:05.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:27:05.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:27:05.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:27:05.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:27:05.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-18T15:27:05.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-18T15:27:05.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:27:05.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:27:05.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:27:05.098 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:27:05.098 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:27:05.099 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:27:05.099 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:27:05.099 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:27:05.099 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:27:05.099 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:27:05.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:27:05.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:27:05.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:27:05.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:27:05.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:27:05.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:27:05.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:27:05.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:27:05.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:27:05.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:27:05.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-18T15:27:05.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-09-18T15:27:05.103 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.9 2024-09-18T15:27:05.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:27:05.104 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-18T15:27:05.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-09-18T15:27:05.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:27:05.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:27:05.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:27:05.136 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:27:05.125+0000 7f1132a4c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:27:05.139 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:27:05.129+0000 7f1132a4c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:27:05.140 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:27:05.129+0000 7f1132a4c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:27:05.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-09-18T15:27:05.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:27:05.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-09-18T15:27:05.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:27:05.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:27:05.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:05.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:27:05.422 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:27:05.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:05.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:27:05.712 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:27:05.701+0000 7f1132a4c540 -1 Falling back to public interface 2024-09-18T15:27:05.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:27:06.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:27:06.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:06.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:27:06.729 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:27:06.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:06.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:27:07.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:27:08.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:27:08.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:08.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:27:08.048 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:27:08.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:08.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:27:08.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:27:08.543 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:27:08.533+0000 7f1132a4c540 -1 osd.9 73 log_to_monitors true 2024-09-18T15:27:09.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:27:09.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:09.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:27:09.365 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:27:09.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:09.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:27:09.839 INFO:tasks.workunit.client.0.smithi046.stderr:osd.9 up in weight 1 up_from 77 up_thru 0 down_at 74 last_clean_interval [50,73) [v2:127.0.0.1:6874/295096,v1:127.0.0.1:6875/295096] [v2:127.0.0.1:6876/295096,v1:127.0.0.1:6877/295096] exists,up 364a066e-2eb8-45f2-a30f-be7da6149b02 2024-09-18T15:27:09.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:27:09.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:27:09.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:27:09.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:27:09.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:27:09.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:27:09.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:27:09.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:27:09.840 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:27:09.840 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:27:09.841 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:27:09.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:27:09.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:27:09.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:27:10.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:27:10.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:27:10.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:27:10.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:27:10.043 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:27:10.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:27:10.400 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:27:10.400 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:27:10.400 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:27:10.400 INFO:tasks.workunit.client.0.smithi046.stderr:4 2024-09-18T15:27:10.400 INFO:tasks.workunit.client.0.smithi046.stderr:5 2024-09-18T15:27:10.400 INFO:tasks.workunit.client.0.smithi046.stderr:6 2024-09-18T15:27:10.400 INFO:tasks.workunit.client.0.smithi046.stderr:7 2024-09-18T15:27:10.400 INFO:tasks.workunit.client.0.smithi046.stderr:8 2024-09-18T15:27:10.400 INFO:tasks.workunit.client.0.smithi046.stderr:9' 2024-09-18T15:27:10.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:27:10.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:10.401 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:27:10.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836569 2024-09-18T15:27:10.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836569 2024-09-18T15:27:10.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836569' 2024-09-18T15:27:10.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:10.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:27:10.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673039 2024-09-18T15:27:10.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673039 2024-09-18T15:27:10.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836569 1-42949673039' 2024-09-18T15:27:10.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:10.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:27:11.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509516 2024-09-18T15:27:11.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509516 2024-09-18T15:27:11.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836569 1-42949673039 2-64424509516' 2024-09-18T15:27:11.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:11.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:27:11.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=309237645320 2024-09-18T15:27:11.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 309237645320 2024-09-18T15:27:11.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836569 1-42949673039 2-64424509516 3-309237645320' 2024-09-18T15:27:11.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:11.315 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:27:11.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182460 2024-09-18T15:27:11.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182460 2024-09-18T15:27:11.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836569 1-42949673039 2-64424509516 3-309237645320 4-107374182460' 2024-09-18T15:27:11.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:11.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:27:11.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018936 2024-09-18T15:27:11.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018936 2024-09-18T15:27:11.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836569 1-42949673039 2-64424509516 3-309237645320 4-107374182460 5-128849018936' 2024-09-18T15:27:11.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:11.703 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:27:11.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855407 2024-09-18T15:27:11.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855407 2024-09-18T15:27:11.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836569 1-42949673039 2-64424509516 3-309237645320 4-107374182460 5-128849018936 6-150323855407' 2024-09-18T15:27:11.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:11.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:27:11.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691879 2024-09-18T15:27:11.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691879 2024-09-18T15:27:11.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836569 1-42949673039 2-64424509516 3-309237645320 4-107374182460 5-128849018936 6-150323855407 7-171798691879' 2024-09-18T15:27:11.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:11.986 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:27:12.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528351 2024-09-18T15:27:12.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528351 2024-09-18T15:27:12.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836569 1-42949673039 2-64424509516 3-309237645320 4-107374182460 5-128849018936 6-150323855407 7-171798691879 8-193273528351' 2024-09-18T15:27:12.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:12.132 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:27:12.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=330712481794 2024-09-18T15:27:12.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 330712481794 2024-09-18T15:27:12.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836569 1-42949673039 2-64424509516 3-309237645320 4-107374182460 5-128849018936 6-150323855407 7-171798691879 8-193273528351 9-330712481794' 2024-09-18T15:27:12.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:12.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836569 2024-09-18T15:27:12.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:12.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:27:12.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836569 2024-09-18T15:27:12.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:12.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836569 2024-09-18T15:27:12.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836569' 2024-09-18T15:27:12.281 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836569 2024-09-18T15:27:12.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:27:12.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836569 -lt 21474836569 2024-09-18T15:27:12.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:12.590 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673039 2024-09-18T15:27:12.590 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:12.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:27:12.593 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673039 2024-09-18T15:27:12.594 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:12.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673039 2024-09-18T15:27:12.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673039' 2024-09-18T15:27:12.596 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 42949673039 2024-09-18T15:27:12.596 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:27:12.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673038 -lt 42949673039 2024-09-18T15:27:12.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:27:13.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:27:13.956 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:27:14.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673039 -lt 42949673039 2024-09-18T15:27:14.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:14.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509516 2024-09-18T15:27:14.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:14.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:27:14.283 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509516 2024-09-18T15:27:14.283 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:14.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509516 2024-09-18T15:27:14.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509516' 2024-09-18T15:27:14.285 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509516 2024-09-18T15:27:14.286 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:27:14.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509516 -lt 64424509516 2024-09-18T15:27:14.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:14.605 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-309237645320 2024-09-18T15:27:14.605 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:14.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:27:14.608 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-309237645320 2024-09-18T15:27:14.609 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:14.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=309237645320 2024-09-18T15:27:14.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 309237645320' 2024-09-18T15:27:14.611 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.3 seq 309237645320 2024-09-18T15:27:14.612 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:27:14.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 309237645320 -lt 309237645320 2024-09-18T15:27:14.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:14.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182460 2024-09-18T15:27:14.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:14.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:27:14.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182460 2024-09-18T15:27:14.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:14.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182460 2024-09-18T15:27:14.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182460' 2024-09-18T15:27:14.956 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.4 seq 107374182460 2024-09-18T15:27:14.957 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:27:15.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182461 -lt 107374182460 2024-09-18T15:27:15.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:15.266 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018936 2024-09-18T15:27:15.266 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:15.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:27:15.270 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018936 2024-09-18T15:27:15.270 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:15.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018936 2024-09-18T15:27:15.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018936' 2024-09-18T15:27:15.273 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.5 seq 128849018936 2024-09-18T15:27:15.274 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:27:15.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018937 -lt 128849018936 2024-09-18T15:27:15.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:15.653 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855407 2024-09-18T15:27:15.653 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:15.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:27:15.657 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855407 2024-09-18T15:27:15.657 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:15.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855407 2024-09-18T15:27:15.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855407' 2024-09-18T15:27:15.659 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.6 seq 150323855407 2024-09-18T15:27:15.660 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:27:16.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855408 -lt 150323855407 2024-09-18T15:27:16.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:16.015 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691879 2024-09-18T15:27:16.015 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:16.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:27:16.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691879 2024-09-18T15:27:16.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:16.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691879 2024-09-18T15:27:16.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691879' 2024-09-18T15:27:16.021 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.7 seq 171798691879 2024-09-18T15:27:16.021 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:27:16.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691880 -lt 171798691879 2024-09-18T15:27:16.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:16.352 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528351 2024-09-18T15:27:16.352 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:16.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:27:16.356 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528351 2024-09-18T15:27:16.356 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:16.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528351 2024-09-18T15:27:16.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528351' 2024-09-18T15:27:16.358 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.8 seq 193273528351 2024-09-18T15:27:16.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:27:16.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528351 -lt 193273528351 2024-09-18T15:27:16.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:16.670 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-330712481794 2024-09-18T15:27:16.670 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:16.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:27:16.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-330712481794 2024-09-18T15:27:16.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:16.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=330712481794 2024-09-18T15:27:16.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 330712481794' 2024-09-18T15:27:16.676 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.9 seq 330712481794 2024-09-18T15:27:16.676 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:27:17.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 330712481795 -lt 330712481794 2024-09-18T15:27:17.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:27:17.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:27:17.032 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:27:17.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:27:17.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:27:17.424 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:27:17.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:27:17.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:27:17.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:27:17.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:27:17.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:27:17.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:27:17.725 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:27:17.726 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:27:17.726 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:27:18.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:27:18.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:27:18.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:27:18.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-18T15:27:18.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-18T15:27:18.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:260: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-09-18T15:27:18.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:225: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-18T15:27:18.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local poolname=ecpool 2024-09-18T15:27:18.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local osd=5 2024-09-18T15:27:18.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-18T15:27:18.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:27:18.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:27:18.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-09-18T15:27:18.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:27:18.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-18T15:27:18.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:27:18.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:27:18.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-09-18T15:27:18.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:27:18.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-18T15:27:18.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:27:18.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:27:18.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:27:18.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:27:18.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:27:18.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:27:18.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-18T15:27:18.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:27:18.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:27:18.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-09-18T15:27:18.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:27:18.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-18T15:27:18.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-18T15:27:20.868 INFO:tasks.workunit.client.0.smithi046.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-18T15:27:21.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-18T15:27:21.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:27:21.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:27:21.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-09-18T15:27:21.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:27:21.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-18T15:27:21.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:27:21.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:27:21.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:27:21.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:27:21.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-18T15:27:21.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-18T15:27:21.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:27:21.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:27:21.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:27:21.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:27:21.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:27:21.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:27:21.410 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:27:21.410 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:27:21.411 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:27:21.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:27:21.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:27:21.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:27:21.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:27:21.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:27:21.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:27:21.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:27:21.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:27:21.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:27:21.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:27:21.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:27:21.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-18T15:27:21.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-09-18T15:27:21.414 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.5 2024-09-18T15:27:21.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:27:21.415 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-18T15:27:21.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-09-18T15:27:21.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:27:21.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:27:21.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:27:21.446 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:27:21.437+0000 7fb85d1fe540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:27:21.449 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:27:21.437+0000 7fb85d1fe540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:27:21.450 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:27:21.441+0000 7fb85d1fe540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:27:21.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-09-18T15:27:21.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:27:21.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T15:27:21.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:27:21.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:27:21.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:21.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:27:21.763 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:27:21.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:21.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:27:22.004 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:27:21.993+0000 7fb85d1fe540 -1 Falling back to public interface 2024-09-18T15:27:22.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:27:23.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:27:23.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:23.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:27:23.071 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:27:23.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:23.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:27:23.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:27:24.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:27:24.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:24.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:27:24.380 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:27:24.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:24.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:27:24.498 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:27:24.489+0000 7fb85d1fe540 -1 osd.5 78 log_to_monitors true 2024-09-18T15:27:24.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:27:25.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:27:25.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:25.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:27:25.821 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:27:25.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:25.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:27:26.451 INFO:tasks.workunit.client.0.smithi046.stderr:osd.5 up in weight 1 up_from 82 up_thru 82 down_at 79 last_clean_interval [30,78) [v2:127.0.0.1:6842/296835,v1:127.0.0.1:6843/296835] [v2:127.0.0.1:6844/296835,v1:127.0.0.1:6845/296835] exists,up 63244125-a488-402d-9bdf-d4f6f2b1fbf0 2024-09-18T15:27:26.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:27:26.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:27:26.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:27:26.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:27:26.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:27:26.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:27:26.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:27:26.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:27:26.452 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:27:26.453 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:27:26.453 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:27:26.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:27:26.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:27:26.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:27:26.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:27:26.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:27:26.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:27:26.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:27:26.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:27:27.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:27:27.358 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:27:27.358 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:27:27.358 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:27:27.358 INFO:tasks.workunit.client.0.smithi046.stderr:4 2024-09-18T15:27:27.359 INFO:tasks.workunit.client.0.smithi046.stderr:5 2024-09-18T15:27:27.359 INFO:tasks.workunit.client.0.smithi046.stderr:6 2024-09-18T15:27:27.359 INFO:tasks.workunit.client.0.smithi046.stderr:7 2024-09-18T15:27:27.359 INFO:tasks.workunit.client.0.smithi046.stderr:8 2024-09-18T15:27:27.359 INFO:tasks.workunit.client.0.smithi046.stderr:9' 2024-09-18T15:27:27.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:27:27.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:27.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:27:27.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836573 2024-09-18T15:27:27.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836573 2024-09-18T15:27:27.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573' 2024-09-18T15:27:27.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:27.535 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:27:27.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673043 2024-09-18T15:27:27.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673043 2024-09-18T15:27:27.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673043' 2024-09-18T15:27:27.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:27.680 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:27:27.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509520 2024-09-18T15:27:27.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509520 2024-09-18T15:27:27.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673043 2-64424509520' 2024-09-18T15:27:27.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:27.820 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:27:27.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=309237645324 2024-09-18T15:27:27.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 309237645324 2024-09-18T15:27:27.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673043 2-64424509520 3-309237645324' 2024-09-18T15:27:27.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:27.967 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:27:28.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182465 2024-09-18T15:27:28.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182465 2024-09-18T15:27:28.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673043 2-64424509520 3-309237645324 4-107374182465' 2024-09-18T15:27:28.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:28.113 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:27:28.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=352187318274 2024-09-18T15:27:28.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 352187318274 2024-09-18T15:27:28.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673043 2-64424509520 3-309237645324 4-107374182465 5-352187318274' 2024-09-18T15:27:28.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:28.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:27:28.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855412 2024-09-18T15:27:28.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855412 2024-09-18T15:27:28.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673043 2-64424509520 3-309237645324 4-107374182465 5-352187318274 6-150323855412' 2024-09-18T15:27:28.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:28.402 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:27:28.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691883 2024-09-18T15:27:28.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691883 2024-09-18T15:27:28.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673043 2-64424509520 3-309237645324 4-107374182465 5-352187318274 6-150323855412 7-171798691883' 2024-09-18T15:27:28.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:28.548 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:27:28.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528355 2024-09-18T15:27:28.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528355 2024-09-18T15:27:28.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673043 2-64424509520 3-309237645324 4-107374182465 5-352187318274 6-150323855412 7-171798691883 8-193273528355' 2024-09-18T15:27:28.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:28.680 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:27:28.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=330712481798 2024-09-18T15:27:28.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 330712481798 2024-09-18T15:27:28.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673043 2-64424509520 3-309237645324 4-107374182465 5-352187318274 6-150323855412 7-171798691883 8-193273528355 9-330712481798' 2024-09-18T15:27:28.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:28.828 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836573 2024-09-18T15:27:28.828 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:28.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:27:28.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836573 2024-09-18T15:27:28.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:28.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836573 2024-09-18T15:27:28.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836573' 2024-09-18T15:27:28.833 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836573 2024-09-18T15:27:28.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:27:29.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836574 -lt 21474836573 2024-09-18T15:27:29.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:29.149 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673043 2024-09-18T15:27:29.149 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:29.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:27:29.152 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673043 2024-09-18T15:27:29.152 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:29.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673043 2024-09-18T15:27:29.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673043' 2024-09-18T15:27:29.154 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 42949673043 2024-09-18T15:27:29.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:27:29.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673044 -lt 42949673043 2024-09-18T15:27:29.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:29.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509520 2024-09-18T15:27:29.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:29.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:27:29.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509520 2024-09-18T15:27:29.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:29.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509520 2024-09-18T15:27:29.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509520' 2024-09-18T15:27:29.514 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509520 2024-09-18T15:27:29.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:27:29.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509520 -lt 64424509520 2024-09-18T15:27:29.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:29.861 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-309237645324 2024-09-18T15:27:29.862 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:29.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:27:29.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:29.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-309237645324 2024-09-18T15:27:29.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=309237645324 2024-09-18T15:27:29.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 309237645324' 2024-09-18T15:27:29.866 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.3 seq 309237645324 2024-09-18T15:27:29.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:27:30.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 309237645324 -lt 309237645324 2024-09-18T15:27:30.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:30.172 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182465 2024-09-18T15:27:30.173 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:30.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:27:30.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182465 2024-09-18T15:27:30.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:30.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182465 2024-09-18T15:27:30.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182465' 2024-09-18T15:27:30.179 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.4 seq 107374182465 2024-09-18T15:27:30.179 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:27:30.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182465 -lt 107374182465 2024-09-18T15:27:30.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:30.527 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-352187318274 2024-09-18T15:27:30.527 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:30.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:27:30.531 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-352187318274 2024-09-18T15:27:30.531 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:30.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=352187318274 2024-09-18T15:27:30.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 352187318274' 2024-09-18T15:27:30.533 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.5 seq 352187318274 2024-09-18T15:27:30.533 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:27:30.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 352187318274 -lt 352187318274 2024-09-18T15:27:30.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:30.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855412 2024-09-18T15:27:30.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:30.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:27:30.892 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855412 2024-09-18T15:27:30.892 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:30.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855412 2024-09-18T15:27:30.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855412' 2024-09-18T15:27:30.894 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.6 seq 150323855412 2024-09-18T15:27:30.895 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:27:31.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855412 -lt 150323855412 2024-09-18T15:27:31.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:31.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691883 2024-09-18T15:27:31.252 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:31.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:27:31.255 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691883 2024-09-18T15:27:31.255 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:31.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691883 2024-09-18T15:27:31.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691883' 2024-09-18T15:27:31.257 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.7 seq 171798691883 2024-09-18T15:27:31.258 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:27:31.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691884 -lt 171798691883 2024-09-18T15:27:31.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:31.666 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528355 2024-09-18T15:27:31.666 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:31.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:27:31.670 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528355 2024-09-18T15:27:31.671 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:31.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528355 2024-09-18T15:27:31.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528355' 2024-09-18T15:27:31.673 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.8 seq 193273528355 2024-09-18T15:27:31.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:27:32.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528356 -lt 193273528355 2024-09-18T15:27:32.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:32.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-330712481798 2024-09-18T15:27:32.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:32.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:27:32.016 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-330712481798 2024-09-18T15:27:32.016 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:32.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=330712481798 2024-09-18T15:27:32.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 330712481798' 2024-09-18T15:27:32.019 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.9 seq 330712481798 2024-09-18T15:27:32.019 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:27:32.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 330712481799 -lt 330712481798 2024-09-18T15:27:32.321 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:27:32.321 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:27:32.322 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:27:32.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:27:32.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:27:32.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:27:32.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:27:32.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:27:32.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:27:32.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:27:32.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:27:33.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:27:33.103 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:27:33.103 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:27:33.104 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:27:33.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:27:33.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:27:33.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:27:33.479 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-18T15:27:33.479 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-09-18T15:27:33.479 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T15:27:33.479 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:27:33.479 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:27:33.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: local pg=2.0 2024-09-18T15:27:33.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: repair 2.0 2024-09-18T15:27:33.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-09-18T15:27:33.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-09-18T15:27:33.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:27:33.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:27:33.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:27:33.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:27:34.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T15:26:55.382803+0000 2024-09-18T15:27:34.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-09-18T15:27:34.454 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-18T15:27:34.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-09-18T15:26:55.382803+0000 2024-09-18T15:27:34.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=2.0 2024-09-18T15:27:34.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:26:55.382803+0000 2024-09-18T15:27:34.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:27:34.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:27:34.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:27:34.463 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:27:34.464 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:27:34.464 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:27:34.464 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:27:34.464 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:27:34.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:26:55.382803+0000 '>' 2024-09-18T15:26:55.382803+0000 2024-09-18T15:27:34.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:27:35.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:27:35.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:27:35.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:27:35.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:27:35.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:27:35.779 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:27:35.779 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:27:36.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:26:55.382803+0000 '>' 2024-09-18T15:26:55.382803+0000 2024-09-18T15:27:36.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:27:37.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:27:37.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:27:37.090 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:27:37.090 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:27:37.090 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:27:37.091 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:27:37.091 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:27:37.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:26:55.382803+0000 '>' 2024-09-18T15:26:55.382803+0000 2024-09-18T15:27:37.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:27:38.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:27:38.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:27:38.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:27:38.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:27:38.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:27:38.409 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:27:38.409 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:27:38.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:26:55.382803+0000 '>' 2024-09-18T15:26:55.382803+0000 2024-09-18T15:27:38.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:27:39.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:27:39.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:27:39.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:27:39.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:27:39.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:27:39.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:27:39.723 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:27:40.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:26:55.382803+0000 '>' 2024-09-18T15:26:55.382803+0000 2024-09-18T15:27:40.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:27:41.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:27:41.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:27:41.037 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:27:41.037 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:27:41.037 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:27:41.037 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:27:41.038 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:27:41.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:26:55.382803+0000 '>' 2024-09-18T15:26:55.382803+0000 2024-09-18T15:27:41.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:27:42.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:27:42.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:27:42.354 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:27:42.354 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:27:42.354 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:27:42.354 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:27:42.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:27:42.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:26:55.382803+0000 '>' 2024-09-18T15:26:55.382803+0000 2024-09-18T15:27:42.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:27:43.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:27:43.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:27:43.669 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:27:43.669 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:27:43.669 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:27:43.669 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:27:43.669 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:27:44.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:26:55.382803+0000 '>' 2024-09-18T15:26:55.382803+0000 2024-09-18T15:27:44.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:27:45.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:27:45.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:27:45.038 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:27:45.038 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:27:45.038 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:27:45.039 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:27:45.039 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:27:45.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:27:43.492924+0000 '>' 2024-09-18T15:26:55.382803+0000 2024-09-18T15:27:45.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:27:45.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:241: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-18T15:27:45.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:27:45.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:27:45.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-09-18T15:27:45.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:27:45.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-18T15:27:45.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:27:45.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:27:45.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-09-18T15:27:45.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:27:45.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-18T15:27:45.353 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:27:45.353 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:27:45.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:27:45.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:27:45.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:27:45.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:27:45.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-18T15:27:45.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:27:45.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:27:45.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-09-18T15:27:45.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:27:45.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-18T15:27:45.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-18T15:27:47.892 INFO:tasks.workunit.client.0.smithi046.stdout:_ 2024-09-18T15:27:47.892 INFO:tasks.workunit.client.0.smithi046.stdout:hinfo_key 2024-09-18T15:27:47.893 INFO:tasks.workunit.client.0.smithi046.stdout:snapset 2024-09-18T15:27:48.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-18T15:27:48.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:27:48.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:27:48.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-09-18T15:27:48.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:27:48.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-18T15:27:48.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:27:48.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:27:48.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:27:48.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:27:48.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-18T15:27:48.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-18T15:27:48.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:27:48.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:27:48.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:27:48.429 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:27:48.430 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:27:48.430 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:27:48.431 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:27:48.431 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:27:48.432 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:27:48.432 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:27:48.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:27:48.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:27:48.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:27:48.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:27:48.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:27:48.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:27:48.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:27:48.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:27:48.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:27:48.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:27:48.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-18T15:27:48.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-09-18T15:27:48.433 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.9 2024-09-18T15:27:48.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:27:48.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-18T15:27:48.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-09-18T15:27:48.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:27:48.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:27:48.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:27:48.464 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:27:48.453+0000 7f6f44342540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:27:48.467 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:27:48.457+0000 7f6f44342540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:27:48.469 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:27:48.457+0000 7f6f44342540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:27:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-09-18T15:27:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:27:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-09-18T15:27:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:27:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:27:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:27:48.751 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:27:48.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:48.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:27:49.004 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:27:48.993+0000 7f6f44342540 -1 Falling back to public interface 2024-09-18T15:27:49.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:27:50.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:27:50.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:50.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:27:50.115 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:27:50.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:50.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:27:50.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:27:51.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:27:51.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:51.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:27:51.419 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:27:51.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:51.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:27:51.766 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:27:51.757+0000 7f6f44342540 -1 osd.9 84 log_to_monitors true 2024-09-18T15:27:51.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:27:52.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:27:52.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:52.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:27:52.875 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:27:52.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:52.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:27:53.330 INFO:tasks.workunit.client.0.smithi046.stderr:osd.9 up in weight 1 up_from 88 up_thru 0 down_at 85 last_clean_interval [77,84) [v2:127.0.0.1:6874/298835,v1:127.0.0.1:6875/298835] [v2:127.0.0.1:6876/298835,v1:127.0.0.1:6877/298835] exists,up 364a066e-2eb8-45f2-a30f-be7da6149b02 2024-09-18T15:27:53.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:27:53.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:27:53.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:27:53.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:27:53.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:27:53.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:27:53.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:27:53.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:27:53.331 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:27:53.331 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:27:53.331 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:27:53.332 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:27:53.332 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:27:53.332 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:27:53.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:27:53.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:27:53.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:27:53.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:27:53.547 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:27:53.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:27:53.866 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:27:53.866 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:27:53.866 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:27:53.866 INFO:tasks.workunit.client.0.smithi046.stderr:4 2024-09-18T15:27:53.866 INFO:tasks.workunit.client.0.smithi046.stderr:5 2024-09-18T15:27:53.866 INFO:tasks.workunit.client.0.smithi046.stderr:6 2024-09-18T15:27:53.867 INFO:tasks.workunit.client.0.smithi046.stderr:7 2024-09-18T15:27:53.867 INFO:tasks.workunit.client.0.smithi046.stderr:8 2024-09-18T15:27:53.867 INFO:tasks.workunit.client.0.smithi046.stderr:9' 2024-09-18T15:27:53.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:27:53.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:53.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:27:54.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836580 2024-09-18T15:27:54.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836580 2024-09-18T15:27:54.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836580' 2024-09-18T15:27:54.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:54.127 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:27:54.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673050 2024-09-18T15:27:54.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673050 2024-09-18T15:27:54.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836580 1-42949673050' 2024-09-18T15:27:54.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:54.454 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:27:54.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509526 2024-09-18T15:27:54.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509526 2024-09-18T15:27:54.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836580 1-42949673050 2-64424509526' 2024-09-18T15:27:54.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:54.606 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:27:54.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=309237645331 2024-09-18T15:27:54.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 309237645331 2024-09-18T15:27:54.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836580 1-42949673050 2-64424509526 3-309237645331' 2024-09-18T15:27:54.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:54.917 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:27:55.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182471 2024-09-18T15:27:55.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182471 2024-09-18T15:27:55.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836580 1-42949673050 2-64424509526 3-309237645331 4-107374182471' 2024-09-18T15:27:55.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:55.067 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:27:55.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=352187318281 2024-09-18T15:27:55.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 352187318281 2024-09-18T15:27:55.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836580 1-42949673050 2-64424509526 3-309237645331 4-107374182471 5-352187318281' 2024-09-18T15:27:55.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:55.229 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:27:55.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855418 2024-09-18T15:27:55.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855418 2024-09-18T15:27:55.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836580 1-42949673050 2-64424509526 3-309237645331 4-107374182471 5-352187318281 6-150323855418' 2024-09-18T15:27:55.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:55.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:27:55.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691890 2024-09-18T15:27:55.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691890 2024-09-18T15:27:55.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836580 1-42949673050 2-64424509526 3-309237645331 4-107374182471 5-352187318281 6-150323855418 7-171798691890' 2024-09-18T15:27:55.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:55.521 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:27:55.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528361 2024-09-18T15:27:55.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528361 2024-09-18T15:27:55.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836580 1-42949673050 2-64424509526 3-309237645331 4-107374182471 5-352187318281 6-150323855418 7-171798691890 8-193273528361' 2024-09-18T15:27:55.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:55.686 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:27:55.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=377957122050 2024-09-18T15:27:55.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 377957122050 2024-09-18T15:27:55.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836580 1-42949673050 2-64424509526 3-309237645331 4-107374182471 5-352187318281 6-150323855418 7-171798691890 8-193273528361 9-377957122050' 2024-09-18T15:27:55.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:55.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836580 2024-09-18T15:27:55.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:55.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:27:55.825 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836580 2024-09-18T15:27:55.825 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:55.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836580 2024-09-18T15:27:55.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836580' 2024-09-18T15:27:55.828 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836580 2024-09-18T15:27:55.828 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:27:56.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836580 -lt 21474836580 2024-09-18T15:27:56.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:56.145 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673050 2024-09-18T15:27:56.145 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:56.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:27:56.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673050 2024-09-18T15:27:56.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:56.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673050 2024-09-18T15:27:56.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673050' 2024-09-18T15:27:56.151 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 42949673050 2024-09-18T15:27:56.151 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:27:56.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673050 -lt 42949673050 2024-09-18T15:27:56.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:56.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509526 2024-09-18T15:27:56.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:56.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:27:56.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509526 2024-09-18T15:27:56.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:56.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509526 2024-09-18T15:27:56.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509526' 2024-09-18T15:27:56.514 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509526 2024-09-18T15:27:56.515 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:27:56.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509526 -lt 64424509526 2024-09-18T15:27:56.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:56.819 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-309237645331 2024-09-18T15:27:56.819 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:56.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:27:56.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-309237645331 2024-09-18T15:27:56.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:56.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=309237645331 2024-09-18T15:27:56.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 309237645331' 2024-09-18T15:27:56.825 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.3 seq 309237645331 2024-09-18T15:27:56.826 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:27:57.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 309237645331 -lt 309237645331 2024-09-18T15:27:57.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:57.203 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182471 2024-09-18T15:27:57.204 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:57.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:27:57.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182471 2024-09-18T15:27:57.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:57.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182471 2024-09-18T15:27:57.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182471' 2024-09-18T15:27:57.210 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.4 seq 107374182471 2024-09-18T15:27:57.210 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:27:57.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182471 -lt 107374182471 2024-09-18T15:27:57.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:57.540 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-352187318281 2024-09-18T15:27:57.541 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:57.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:27:57.544 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-352187318281 2024-09-18T15:27:57.544 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:57.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=352187318281 2024-09-18T15:27:57.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 352187318281' 2024-09-18T15:27:57.546 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.5 seq 352187318281 2024-09-18T15:27:57.547 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:27:57.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 352187318281 -lt 352187318281 2024-09-18T15:27:57.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:57.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855418 2024-09-18T15:27:57.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:57.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:27:57.905 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855418 2024-09-18T15:27:57.905 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:57.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855418 2024-09-18T15:27:57.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855418' 2024-09-18T15:27:57.907 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.6 seq 150323855418 2024-09-18T15:27:57.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:27:58.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855418 -lt 150323855418 2024-09-18T15:27:58.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:58.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691890 2024-09-18T15:27:58.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:58.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:27:58.221 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691890 2024-09-18T15:27:58.221 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:58.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691890 2024-09-18T15:27:58.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691890' 2024-09-18T15:27:58.223 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.7 seq 171798691890 2024-09-18T15:27:58.223 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:27:58.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691890 -lt 171798691890 2024-09-18T15:27:58.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:58.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528361 2024-09-18T15:27:58.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:58.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:27:58.584 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528361 2024-09-18T15:27:58.585 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:58.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528361 2024-09-18T15:27:58.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528361' 2024-09-18T15:27:58.587 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.8 seq 193273528361 2024-09-18T15:27:58.587 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:27:58.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528362 -lt 193273528361 2024-09-18T15:27:58.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:58.896 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-377957122050 2024-09-18T15:27:58.897 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:58.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:27:58.900 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-377957122050 2024-09-18T15:27:58.900 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:58.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=377957122050 2024-09-18T15:27:58.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 377957122050' 2024-09-18T15:27:58.903 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.9 seq 377957122050 2024-09-18T15:27:58.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:27:59.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 377957122051 -lt 377957122050 2024-09-18T15:27:59.290 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:27:59.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:27:59.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:27:59.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:27:59.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:27:59.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:27:59.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:27:59.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:27:59.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:27:59.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:27:59.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:28:00.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:28:00.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:28:00.023 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:28:00.023 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:28:00.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:28:00.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:28:00.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:28:00.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-18T15:28:00.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-18T15:28:00.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-09-18T15:28:00.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:184: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-18T15:28:00.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local poolname=ecpool 2024-09-18T15:28:00.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local first=5 2024-09-18T15:28:00.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local second=9 2024-09-18T15:28:00.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:192: corrupt_and_repair_two: pids= 2024-09-18T15:28:00.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-18T15:28:00.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:28:00.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:28:00.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 300140"' 2024-09-18T15:28:00.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 300140' 2024-09-18T15:28:00.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-09-18T15:28:00.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:28:00.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:28:00.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 300141"' 2024-09-18T15:28:00.516 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 300141' 2024-09-18T15:28:00.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: wait_background pids 2024-09-18T15:28:00.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: wait_background: pids=' 300140 300141' 2024-09-18T15:28:00.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: wait_background: return_code=0 2024-09-18T15:28:00.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:28:00.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 300140 2024-09-18T15:28:00.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-18T15:28:00.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/300143: /' 2024-09-18T15:28:00.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-09-18T15:28:00.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/300145: /' 2024-09-18T15:28:04.823 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:28:04.824 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:28:04.824 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-09-18T15:28:04.824 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:28:04.824 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-18T15:28:04.825 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:28:04.825 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:28:04.825 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-09-18T15:28:04.825 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:28:04.825 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-18T15:28:04.825 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:28:04.826 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:28:04.826 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:28:04.826 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:28:04.826 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:28:04.826 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:28:04.827 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-18T15:28:04.827 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:28:04.827 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:28:04.827 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-09-18T15:28:04.827 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:28:04.828 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-18T15:28:04.828 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-18T15:28:04.828 INFO:tasks.workunit.client.0.smithi046.stderr:300143: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-18T15:28:04.828 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-18T15:28:04.828 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:28:04.829 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:28:04.829 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-09-18T15:28:04.829 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:28:04.829 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-18T15:28:04.829 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:28:04.829 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:28:04.830 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:28:04.830 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:28:04.831 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-18T15:28:04.831 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-18T15:28:04.831 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:28:04.831 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:28:04.831 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:28:04.831 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:28:04.832 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:28:04.832 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:28:04.832 INFO:tasks.workunit.client.0.smithi046.stderr:300143: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:28:04.832 INFO:tasks.workunit.client.0.smithi046.stderr:300143: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:28:04.833 INFO:tasks.workunit.client.0.smithi046.stderr:300143: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:28:04.833 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:28:04.833 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:28:04.833 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:28:04.833 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:28:04.834 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:28:04.834 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:28:04.834 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:28:04.834 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:28:04.834 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:28:04.834 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:28:04.835 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:28:04.835 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-18T15:28:04.835 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-09-18T15:28:04.835 INFO:tasks.workunit.client.0.smithi046.stderr:300143: start osd.5 2024-09-18T15:28:04.957 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:28:04.957 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:28:04.957 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-09-18T15:28:04.957 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:28:04.958 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-09-18T15:28:04.958 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:28:04.958 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:28:04.958 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-09-18T15:28:04.958 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:28:04.958 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-18T15:28:04.958 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:28:04.958 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:28:04.958 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:28:04.958 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:28:04.958 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:28:04.958 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:28:04.958 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-09-18T15:28:04.959 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:28:04.959 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:28:04.959 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-09-18T15:28:04.959 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:28:04.959 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-18T15:28:04.959 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-09-18T15:28:04.959 INFO:tasks.workunit.client.0.smithi046.stderr:300145: remove 2#2:eb822e21:::SOMETHING:head# 2024-09-18T15:28:04.959 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-18T15:28:04.959 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:28:04.959 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:28:04.959 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-09-18T15:28:04.959 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:28:04.959 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-18T15:28:04.960 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:28:04.960 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:28:04.962 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:28:04.962 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:28:04.962 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-18T15:28:04.962 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-18T15:28:04.962 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:28:04.963 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:28:04.963 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:28:04.963 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:28:04.963 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:28:04.963 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:28:04.963 INFO:tasks.workunit.client.0.smithi046.stderr:300145: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:28:04.963 INFO:tasks.workunit.client.0.smithi046.stderr:300145: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:28:04.963 INFO:tasks.workunit.client.0.smithi046.stderr:300145: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:28:04.963 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:28:04.963 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:28:04.963 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:28:04.963 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:28:04.963 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:28:04.964 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:28:04.964 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:28:04.964 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:28:04.964 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:28:04.964 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:28:04.964 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:28:04.964 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-18T15:28:04.964 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-09-18T15:28:04.964 INFO:tasks.workunit.client.0.smithi046.stderr:300145: start osd.9 2024-09-18T15:28:09.624 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:28:09.624 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-18T15:28:09.624 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-09-18T15:28:09.625 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:28:09.625 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:28:09.625 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:28:09.625 INFO:tasks.workunit.client.0.smithi046.stderr:300143: 2024-09-18T15:28:04.853+0000 7ffbea54d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:09.625 INFO:tasks.workunit.client.0.smithi046.stderr:300143: 2024-09-18T15:28:04.857+0000 7ffbea54d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:09.625 INFO:tasks.workunit.client.0.smithi046.stderr:300143: 2024-09-18T15:28:04.861+0000 7ffbea54d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:09.625 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-09-18T15:28:09.625 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:28:09.625 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T15:28:09.625 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:28:09.625 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:28:09.625 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:09.626 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:28:09.626 INFO:tasks.workunit.client.0.smithi046.stderr:300143: 0 2024-09-18T15:28:09.626 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:09.626 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:28:09.626 INFO:tasks.workunit.client.0.smithi046.stderr:300143: 2024-09-18T15:28:05.413+0000 7ffbea54d540 -1 Falling back to public interface 2024-09-18T15:28:09.626 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:09.626 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:09.626 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:09.626 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:28:09.626 INFO:tasks.workunit.client.0.smithi046.stderr:300143: 1 2024-09-18T15:28:09.626 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:09.626 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:28:09.626 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:09.627 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:09.627 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:09.627 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:28:09.627 INFO:tasks.workunit.client.0.smithi046.stderr:300143: 2 2024-09-18T15:28:09.627 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:09.627 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:28:09.627 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:09.627 INFO:tasks.workunit.client.0.smithi046.stderr:300143: 2024-09-18T15:28:08.649+0000 7ffbea54d540 -1 osd.5 90 log_to_monitors true 2024-09-18T15:28:09.627 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:09.627 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:09.627 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:28:09.627 INFO:tasks.workunit.client.0.smithi046.stderr:300143: 3 2024-09-18T15:28:09.627 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:09.628 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:28:09.641 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-heops 2024-09-18T15:28:09.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-18T15:28:09.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-09-18T15:28:09.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:28:09.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:28:09.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:28:09.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: 2024-09-18T15:28:04.981+0000 7f17f69c2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:09.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: 2024-09-18T15:28:04.993+0000 7f17f69c2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:09.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: 2024-09-18T15:28:04.997+0000 7f17f69c2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:09.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-09-18T15:28:09.642 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:28:09.642 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-09-18T15:28:09.642 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:28:09.642 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:28:09.642 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:09.642 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:28:09.642 INFO:tasks.workunit.client.0.smithi046.stderr:300145: 0 2024-09-18T15:28:09.642 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:09.642 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:28:09.642 INFO:tasks.workunit.client.0.smithi046.stderr:300145: 2024-09-18T15:28:05.537+0000 7f17f69c2540 -1 Falling back to public interface 2024-09-18T15:28:09.642 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:09.642 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:09.642 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:09.642 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:28:09.643 INFO:tasks.workunit.client.0.smithi046.stderr:300145: 1 2024-09-18T15:28:09.643 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:09.643 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:28:09.643 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:09.643 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:09.643 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:09.643 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:28:09.643 INFO:tasks.workunit.client.0.smithi046.stderr:300145: 2 2024-09-18T15:28:09.643 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:09.643 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:28:09.643 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:09.643 INFO:tasks.workunit.client.0.smithi046.stderr:300145: 2024-09-18T15:28:09.073+0000 7f17f69c2540 -1 osd.9 90 log_to_monitors true 2024-09-18T15:28:09.643 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:09.644 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:09.644 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:28:09.644 INFO:tasks.workunit.client.0.smithi046.stderr:300145: 3 2024-09-18T15:28:09.644 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:09.644 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:28:12.380 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:12.380 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:12.381 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:12.381 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:28:12.381 INFO:tasks.workunit.client.0.smithi046.stderr:300143: 4 2024-09-18T15:28:12.381 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:12.381 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:28:12.381 INFO:tasks.workunit.client.0.smithi046.stderr:300143: osd.5 up in weight 1 up_from 96 up_thru 96 down_at 91 last_clean_interval [82,90) [v2:127.0.0.1:6842/300953,v1:127.0.0.1:6843/300953] [v2:127.0.0.1:6844/300953,v1:127.0.0.1:6845/300953] exists,up 63244125-a488-402d-9bdf-d4f6f2b1fbf0 2024-09-18T15:28:12.381 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:28:12.381 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:28:12.381 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:28:12.382 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:28:12.382 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:28:12.382 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:28:12.382 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:28:12.382 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:28:12.382 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:28:12.382 INFO:tasks.workunit.client.0.smithi046.stderr:300143: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:28:12.382 INFO:tasks.workunit.client.0.smithi046.stderr:300143: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:28:12.382 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:28:12.383 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:28:12.383 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:28:12.383 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:28:12.383 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:28:12.383 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:28:12.383 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:28:12.383 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:28:12.383 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:28:12.383 INFO:tasks.workunit.client.0.smithi046.stderr:300143: 1 2024-09-18T15:28:12.383 INFO:tasks.workunit.client.0.smithi046.stderr:300143: 2 2024-09-18T15:28:12.383 INFO:tasks.workunit.client.0.smithi046.stderr:300143: 3 2024-09-18T15:28:12.384 INFO:tasks.workunit.client.0.smithi046.stderr:300143: 4 2024-09-18T15:28:12.384 INFO:tasks.workunit.client.0.smithi046.stderr:300143: 5 2024-09-18T15:28:12.384 INFO:tasks.workunit.client.0.smithi046.stderr:300143: 6 2024-09-18T15:28:12.384 INFO:tasks.workunit.client.0.smithi046.stderr:300143: 7 2024-09-18T15:28:12.384 INFO:tasks.workunit.client.0.smithi046.stderr:300143: 8 2024-09-18T15:28:12.384 INFO:tasks.workunit.client.0.smithi046.stderr:300143: 9' 2024-09-18T15:28:12.384 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:28:12.385 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:12.385 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:28:12.385 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836584 2024-09-18T15:28:12.385 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836584 2024-09-18T15:28:12.385 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836584' 2024-09-18T15:28:12.386 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:12.386 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/lpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:12.386 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:12.387 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:12.387 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:28:12.387 INFO:tasks.workunit.client.0.smithi046.stderr:300145: 4 2024-09-18T15:28:12.387 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:12.387 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:28:12.387 INFO:tasks.workunit.client.0.smithi046.stderr:300145: osd.9 up in weight 1 up_from 97 up_thru 0 down_at 91 last_clean_interval [88,90) [v2:127.0.0.1:6874/300998,v1:127.0.0.1:6875/300998] [v2:127.0.0.1:6876/300998,v1:127.0.0.1:6877/300998] exists,up 364a066e-2eb8-45f2-a30f-be7da6149b02 2024-09-18T15:28:12.388 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:28:12.388 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:28:12.388 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:28:12.388 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:28:12.388 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:28:12.388 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:28:12.388 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:28:12.388 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:28:12.388 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:28:12.389 INFO:tasks.workunit.client.0.smithi046.stderr:300145: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:28:12.389 INFO:tasks.workunit.client.0.smithi046.stderr:300145: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:28:12.389 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:28:12.389 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:28:12.389 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:28:12.389 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:28:12.389 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:28:12.390 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:28:12.390 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:28:12.390 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:28:12.390 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:28:12.390 INFO:tasks.workunit.client.0.smithi046.stderr:300145: 1 2024-09-18T15:28:12.390 INFO:tasks.workunit.client.0.smithi046.stderr:300145: 2 2024-09-18T15:28:12.390 INFO:tasks.workunit.client.0.smithi046.stderr:300145: 3 2024-09-18T15:28:12.390 INFO:tasks.workunit.client.0.smithi046.stderr:300145: 4 2024-09-18T15:28:12.390 INFO:tasks.workunit.client.0.smithi046.stderr:300145: 5 2024-09-18T15:28:12.390 INFO:tasks.workunit.client.0.smithi046.stderr:300145: 6 2024-09-18T15:28:12.390 INFO:tasks.workunit.client.0.smithi046.stderr:300145: 7 2024-09-18T15:28:12.390 INFO:tasks.workunit.client.0.smithi046.stderr:300145: 8 2024-09-18T15:28:12.391 INFO:tasks.workunit.client.0.smithi046.stderr:300145: 9' 2024-09-18T15:28:12.391 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:28:12.391 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:12.391 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:28:12.391 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836585 2024-09-18T15:28:12.391 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836585 2024-09-18T15:28:12.391 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836585' 2024-09-18T15:28:12.391 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:13.753 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:28:13.753 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673055 2024-09-18T15:28:13.753 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673055 2024-09-18T15:28:13.753 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836585 1-42949673055' 2024-09-18T15:28:13.754 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:13.754 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:28:13.754 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509531 2024-09-18T15:28:13.754 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509531 2024-09-18T15:28:13.754 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836585 1-42949673055 2-64424509531' 2024-09-18T15:28:13.754 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:13.754 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:28:13.754 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=309237645336 2024-09-18T15:28:13.754 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 309237645336 2024-09-18T15:28:13.754 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836585 1-42949673055 2-64424509531 3-309237645336' 2024-09-18T15:28:13.754 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:13.755 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:28:13.755 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182476 2024-09-18T15:28:13.755 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182476 2024-09-18T15:28:13.755 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836585 1-42949673055 2-64424509531 3-309237645336 4-107374182476' 2024-09-18T15:28:13.755 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:13.755 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:28:13.755 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=412316860419 2024-09-18T15:28:13.755 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 412316860419 2024-09-18T15:28:13.755 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836585 1-42949673055 2-64424509531 3-309237645336 4-107374182476 5-412316860419' 2024-09-18T15:28:13.755 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:13.755 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:28:13.755 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855423 2024-09-18T15:28:13.755 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855423 2024-09-18T15:28:13.756 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836585 1-42949673055 2-64424509531 3-309237645336 4-107374182476 5-412316860419 6-150323855423' 2024-09-18T15:28:13.756 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:13.756 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:28:13.756 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691894 2024-09-18T15:28:13.759 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /hqa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:28:13.760 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673054 2024-09-18T15:28:13.760 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673054 2024-09-18T15:28:13.760 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836584 1-42949673054' 2024-09-18T15:28:13.760 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:13.760 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:28:13.760 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509532 2024-09-18T15:28:13.760 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509532 2024-09-18T15:28:13.760 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836584 1-42949673054 2-64424509532' 2024-09-18T15:28:13.760 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:13.760 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:28:13.760 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=309237645335 2024-09-18T15:28:13.760 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 309237645335 2024-09-18T15:28:13.761 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836584 1-42949673054 2-64424509532 3-309237645335' 2024-09-18T15:28:13.761 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:13.761 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:28:13.761 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182477 2024-09-18T15:28:13.761 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182477 2024-09-18T15:28:13.761 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836584 1-42949673054 2-64424509532 3-309237645335 4-107374182477' 2024-09-18T15:28:13.761 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:13.761 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:28:13.761 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=412316860418 2024-09-18T15:28:13.761 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 412316860418 2024-09-18T15:28:13.761 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836584 1-42949673054 2-64424509532 3-309237645335 4-107374182477 5-412316860418' 2024-09-18T15:28:13.761 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:13.762 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:28:13.762 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855424 2024-09-18T15:28:13.762 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855424 2024-09-18T15:28:13.762 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836584 1-42949673054 2-64424509532 3-309237645335 4-107374182477 5-412316860418 6-150323855424' 2024-09-18T15:28:13.762 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:13.762 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:28:13.762 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691895 2024-09-18T15:28:14.654 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /ome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691894 2024-09-18T15:28:14.654 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836585 1-42949673055 2-64424509531 3-309237645336 4-107374182476 5-412316860419 6-150323855423 7-171798691894' 2024-09-18T15:28:14.654 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:14.654 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:28:14.654 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528366 2024-09-18T15:28:14.654 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528366 2024-09-18T15:28:14.655 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836585 1-42949673055 2-64424509531 3-309237645336 4-107374182476 5-412316860419 6-150323855423 7-171798691894 8-193273528366' 2024-09-18T15:28:14.655 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:14.655 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:28:14.655 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=416611827714 2024-09-18T15:28:14.655 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 416611827714 2024-09-18T15:28:14.655 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836585 1-42949673055 2-64424509531 3-309237645336 4-107374182476 5-412316860419 6-150323855423 7-171798691894 8-193273528366 9-416611827714' 2024-09-18T15:28:14.655 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:14.655 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:14.655 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836585 2024-09-18T15:28:14.655 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:28:14.655 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836585 2024-09-18T15:28:14.656 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:14.656 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836585 2024-09-18T15:28:14.656 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836585' 2024-09-18T15:28:14.656 INFO:tasks.workunit.client.0.smithi046.stderr:300145: waiting osd.0 seq 21474836585 2024-09-18T15:28:14.656 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:28:14.656 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836585 -lt 21474836585 2024-09-18T15:28:14.656 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:14.656 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673055 2024-09-18T15:28:14.656 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:14.656 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:28:14.656 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673055 2024-09-18T15:28:14.656 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:14.656 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673055 2024-09-18T15:28:14.657 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673055' 2024-09-18T15:28:14.657 INFO:tasks.workunit.client.0.smithi046.stderr:300145: waiting osd.1 seq 42949673055 2024-09-18T15:28:14.657 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:28:14.659 INFO:tasks.workunit.client.0.smithi046.stderr:300145home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691895 2024-09-18T15:28:14.659 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836584 1-42949673054 2-64424509532 3-309237645335 4-107374182477 5-412316860418 6-150323855424 7-171798691895' 2024-09-18T15:28:14.659 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:14.659 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:28:14.660 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528367 2024-09-18T15:28:14.660 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528367 2024-09-18T15:28:14.660 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836584 1-42949673054 2-64424509532 3-309237645335 4-107374182477 5-412316860418 6-150323855424 7-171798691895 8-193273528367' 2024-09-18T15:28:14.660 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:14.660 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:28:14.660 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=416611827715 2024-09-18T15:28:14.660 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 416611827715 2024-09-18T15:28:14.660 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836584 1-42949673054 2-64424509532 3-309237645335 4-107374182477 5-412316860418 6-150323855424 7-171798691895 8-193273528367 9-416611827715' 2024-09-18T15:28:14.660 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:14.660 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836584 2024-09-18T15:28:14.660 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:14.660 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:28:14.661 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836584 2024-09-18T15:28:14.661 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:14.661 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836584 2024-09-18T15:28:14.661 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836584' 2024-09-18T15:28:14.661 INFO:tasks.workunit.client.0.smithi046.stderr:300143: waiting osd.0 seq 21474836584 2024-09-18T15:28:14.661 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:28:14.661 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836585 -lt 21474836584 2024-09-18T15:28:14.661 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:14.661 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:14.661 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673054 2024-09-18T15:28:14.661 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:28:14.661 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673054 2024-09-18T15:28:14.661 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:14.662 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673054 2024-09-18T15:28:14.662 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673054' 2024-09-18T15:28:14.662 INFO:tasks.workunit.client.0.smithi046.stderr:300143: waiting osd.1 seq 42949673054 2024-09-18T15:28:14.662 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:28:15.639 INFO:tasks.workunit.client.0.smithi046.stderr:30014: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673055 -lt 42949673055 2024-09-18T15:28:15.639 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:15.639 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509531 2024-09-18T15:28:15.639 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:15.639 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:28:15.639 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509531 2024-09-18T15:28:15.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:15.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509531 2024-09-18T15:28:15.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509531' 2024-09-18T15:28:15.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: waiting osd.2 seq 64424509531 2024-09-18T15:28:15.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:28:15.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509532 -lt 64424509531 2024-09-18T15:28:15.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:15.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-309237645336 2024-09-18T15:28:15.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:15.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:28:15.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-309237645336 2024-09-18T15:28:15.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:15.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=309237645336 2024-09-18T15:28:15.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 309237645336' 2024-09-18T15:28:15.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: waiting osd.3 seq 309237645336 2024-09-18T15:28:15.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:28:15.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 309237645337 -lt 309237645336 2024-09-18T15:28:15.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:15.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182476 2024-09-18T15:28:15.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:15.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:28:15.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182476 2024-09-18T15:28:15.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:15.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182476 2024-09-18T15:28:15.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182476' 2024-09-18T15:28:15.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: waiting osd.4 seq 107374182476 2024-09-18T15:28:15.642 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:28:15.642 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182477 -lt 107374182476 2024-09-18T15:28:15.642 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:15.642 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-412316860419 2024-09-18T15:28:15.719 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ce3: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673055 -lt 42949673054 2024-09-18T15:28:15.719 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:15.719 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:15.719 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509532 2024-09-18T15:28:15.719 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:28:15.719 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509532 2024-09-18T15:28:15.719 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:15.719 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509532 2024-09-18T15:28:15.719 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509532' 2024-09-18T15:28:15.720 INFO:tasks.workunit.client.0.smithi046.stderr:300143: waiting osd.2 seq 64424509532 2024-09-18T15:28:15.720 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:28:15.720 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509532 -lt 64424509532 2024-09-18T15:28:15.720 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:15.720 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-309237645335 2024-09-18T15:28:15.720 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:15.720 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:28:15.720 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-309237645335 2024-09-18T15:28:15.720 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:15.720 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=309237645335 2024-09-18T15:28:15.720 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 309237645335' 2024-09-18T15:28:15.720 INFO:tasks.workunit.client.0.smithi046.stderr:300143: waiting osd.3 seq 309237645335 2024-09-18T15:28:15.720 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:28:15.720 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 309237645337 -lt 309237645335 2024-09-18T15:28:15.721 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:15.721 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182477 2024-09-18T15:28:15.721 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:15.721 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:28:15.721 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:15.721 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182477 2024-09-18T15:28:15.721 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182477 2024-09-18T15:28:15.721 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182477' 2024-09-18T15:28:15.721 INFO:tasks.workunit.client.0.smithi046.stderr:300143: waiting osd.4 seq 107374182477 2024-09-18T15:28:15.721 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:28:15.721 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182477 -lt 107374182477 2024-09-18T15:28:15.722 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:15.722 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:16.638 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-heph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:16.638 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:28:16.638 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-412316860419 2024-09-18T15:28:16.638 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:16.639 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=412316860419 2024-09-18T15:28:16.639 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 412316860419' 2024-09-18T15:28:16.639 INFO:tasks.workunit.client.0.smithi046.stderr:300145: waiting osd.5 seq 412316860419 2024-09-18T15:28:16.639 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:28:16.639 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 412316860420 -lt 412316860419 2024-09-18T15:28:16.639 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:16.639 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855423 2024-09-18T15:28:16.639 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:16.639 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:28:16.639 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855423 2024-09-18T15:28:16.639 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:16.639 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855423 2024-09-18T15:28:16.639 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855423' 2024-09-18T15:28:16.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: waiting osd.6 seq 150323855423 2024-09-18T15:28:16.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:28:16.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855424 -lt 150323855423 2024-09-18T15:28:16.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:16.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691894 2024-09-18T15:28:16.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:16.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:28:16.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691894 2024-09-18T15:28:16.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:16.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691894 2024-09-18T15:28:16.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691894' 2024-09-18T15:28:16.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: waiting osd.7 seq 171798691894 2024-09-18T15:28:16.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:28:16.640 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691896 -lt 171798691894 2024-09-18T15:28:16.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:16.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528366 2024-09-18T15:28:16.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:16.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:28:16.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528366 2024-09-18T15:28:16.641 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:16.677 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephlpers.sh:2182: flush_pg_stats: echo 5-412316860418 2024-09-18T15:28:16.677 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:28:16.677 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-412316860418 2024-09-18T15:28:16.677 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:16.677 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=412316860418 2024-09-18T15:28:16.677 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 412316860418' 2024-09-18T15:28:16.678 INFO:tasks.workunit.client.0.smithi046.stderr:300143: waiting osd.5 seq 412316860418 2024-09-18T15:28:16.678 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:28:16.678 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 412316860420 -lt 412316860418 2024-09-18T15:28:16.678 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:16.678 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855424 2024-09-18T15:28:16.678 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:16.678 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:28:16.678 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855424 2024-09-18T15:28:16.678 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:16.678 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855424 2024-09-18T15:28:16.678 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855424' 2024-09-18T15:28:16.679 INFO:tasks.workunit.client.0.smithi046.stderr:300143: waiting osd.6 seq 150323855424 2024-09-18T15:28:16.679 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:28:16.679 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855424 -lt 150323855424 2024-09-18T15:28:16.679 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:16.679 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691895 2024-09-18T15:28:16.679 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:16.679 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:28:16.679 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691895 2024-09-18T15:28:16.679 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:16.680 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691895 2024-09-18T15:28:16.680 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691895' 2024-09-18T15:28:16.680 INFO:tasks.workunit.client.0.smithi046.stderr:300143: waiting osd.7 seq 171798691895 2024-09-18T15:28:16.680 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:28:16.680 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691896 -lt 171798691895 2024-09-18T15:28:16.680 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:16.680 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528367 2024-09-18T15:28:16.681 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:16.681 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:28:16.681 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528367 2024-09-18T15:28:16.681 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:18.455 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528367 2024-09-18T15:28:18.455 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528367' 2024-09-18T15:28:18.455 INFO:tasks.workunit.client.0.smithi046.stderr:300143: waiting osd.8 seq 193273528367 2024-09-18T15:28:18.455 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:28:18.455 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528367 -lt 193273528367 2024-09-18T15:28:18.455 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:18.455 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-416611827715 2024-09-18T15:28:18.455 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:18.455 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:28:18.456 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-416611827715 2024-09-18T15:28:18.456 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:18.456 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=416611827715 2024-09-18T15:28:18.456 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 416611827715' 2024-09-18T15:28:18.456 INFO:tasks.workunit.client.0.smithi046.stderr:300143: waiting osd.9 seq 416611827715 2024-09-18T15:28:18.456 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:28:18.456 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 416611827716 -lt 416611827715 2024-09-18T15:28:18.456 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:28:18.456 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:28:18.456 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:28:18.456 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:28:18.456 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:28:18.457 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:28:18.457 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:28:18.457 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:28:18.457 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:28:18.457 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:28:18.457 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:28:18.457 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:28:18.457 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:28:18.457 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:28:18.457 INFO:tasks.workunit.client.0.smithi046.stderr:300143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:28:18.457 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:28:18.457 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:28:18.457 INFO:tasks.workunit.client.0.smithi046.stderr:300143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:28:18.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:28:18.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:28:18.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 300141 2024-09-18T15:28:18.458 INFO:tasks.workunit.client.0.smithi046.stderr:test/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528366 2024-09-18T15:28:18.458 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528366' 2024-09-18T15:28:18.458 INFO:tasks.workunit.client.0.smithi046.stderr:300145: waiting osd.8 seq 193273528366 2024-09-18T15:28:18.459 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:28:18.459 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528367 -lt 193273528366 2024-09-18T15:28:18.459 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:18.459 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-416611827714 2024-09-18T15:28:18.459 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:18.459 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:28:18.459 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-416611827714 2024-09-18T15:28:18.459 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:18.459 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=416611827714 2024-09-18T15:28:18.459 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 416611827714' 2024-09-18T15:28:18.459 INFO:tasks.workunit.client.0.smithi046.stderr:300145: waiting osd.9 seq 416611827714 2024-09-18T15:28:18.459 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:28:18.459 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 416611827716 -lt 416611827714 2024-09-18T15:28:18.459 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:28:18.460 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:28:18.460 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:28:18.460 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:28:18.460 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:28:18.460 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:28:18.460 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:28:18.460 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:28:18.460 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:28:18.460 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:28:18.460 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:28:18.460 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:28:18.460 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:28:18.460 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:28:18.461 INFO:tasks.workunit.client.0.smithi046.stderr:300145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:28:18.461 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:28:18.461 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:28:18.461 INFO:tasks.workunit.client.0.smithi046.stderr:300145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:28:18.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:28:18.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: eval 'pids='\'''\''' 2024-09-18T15:28:18.462 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: pids= 2024-09-18T15:28:18.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: wait_background: return 0 2024-09-18T15:28:18.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: return_code=0 2024-09-18T15:28:18.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-18T15:28:18.462 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-18T15:28:18.462 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-09-18T15:28:18.462 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T15:28:18.463 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:28:18.463 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:28:18.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: local pg=2.0 2024-09-18T15:28:18.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: repair 2.0 2024-09-18T15:28:18.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-09-18T15:28:18.818 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-09-18T15:28:18.819 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:28:18.819 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:28:18.819 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:28:18.819 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:28:19.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T15:27:43.492924+0000 2024-09-18T15:28:19.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-09-18T15:28:19.464 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-18T15:28:19.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-09-18T15:27:43.492924+0000 2024-09-18T15:28:19.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=2.0 2024-09-18T15:28:19.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:27:43.492924+0000 2024-09-18T15:28:19.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:28:19.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:28:19.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:28:19.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:28:19.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:28:19.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:28:19.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:28:19.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:28:19.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:27:43.492924+0000 '>' 2024-09-18T15:27:43.492924+0000 2024-09-18T15:28:19.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:28:20.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:28:20.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:28:20.786 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:28:20.786 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:28:20.786 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:28:20.787 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:28:20.787 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:28:21.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:27:43.492924+0000 '>' 2024-09-18T15:27:43.492924+0000 2024-09-18T15:28:21.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:28:22.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:28:22.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:28:22.103 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:28:22.103 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:28:22.103 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:28:22.103 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:28:22.104 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:28:22.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:27:43.492924+0000 '>' 2024-09-18T15:27:43.492924+0000 2024-09-18T15:28:22.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:28:23.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:28:23.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:28:23.431 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:28:23.432 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:28:23.432 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:28:23.432 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:28:23.432 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:28:23.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:27:43.492924+0000 '>' 2024-09-18T15:27:43.492924+0000 2024-09-18T15:28:23.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:28:24.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:28:24.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:28:24.756 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:28:24.756 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:28:24.756 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:28:24.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:28:24.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:28:25.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:28:22.471989+0000 '>' 2024-09-18T15:27:43.492924+0000 2024-09-18T15:28:25.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:28:25.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:207: corrupt_and_repair_two: pids= 2024-09-18T15:28:25.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-18T15:28:25.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:28:25.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:28:25.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 303789"' 2024-09-18T15:28:25.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 303789' 2024-09-18T15:28:25.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-18T15:28:25.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:28:25.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:28:25.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-18T15:28:25.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 303790"' 2024-09-18T15:28:25.076 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 303790' 2024-09-18T15:28:25.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: wait_background pids 2024-09-18T15:28:25.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: wait_background: pids=' 303789 303790' 2024-09-18T15:28:25.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: wait_background: return_code=0 2024-09-18T15:28:25.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:28:25.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 303789 2024-09-18T15:28:25.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/303792: /' 2024-09-18T15:28:25.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-18T15:28:25.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/303795: /' 2024-09-18T15:28:28.831 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:28:28.831 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:28:28.832 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-09-18T15:28:28.832 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:28:28.832 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-18T15:28:28.832 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:28:28.832 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:28:28.832 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-09-18T15:28:28.832 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:28:28.832 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-18T15:28:28.832 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:28:28.832 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:28:28.832 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:28:28.832 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:28:28.832 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:28:28.833 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:28:28.833 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-18T15:28:28.833 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:28:28.833 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:28:28.833 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-09-18T15:28:28.833 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:28:28.833 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-18T15:28:28.833 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-09-18T15:28:28.833 INFO:tasks.workunit.client.0.smithi046.stderr:303792: _ 2024-09-18T15:28:28.833 INFO:tasks.workunit.client.0.smithi046.stderr:303792: hinfo_key 2024-09-18T15:28:28.833 INFO:tasks.workunit.client.0.smithi046.stderr:303792: snapset 2024-09-18T15:28:28.833 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-18T15:28:28.833 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:28:28.833 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:28:28.834 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-09-18T15:28:28.834 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:28:28.834 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-18T15:28:28.834 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:28:28.834 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:28:28.837 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:28:28.837 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:28:28.837 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-18T15:28:28.837 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-18T15:28:28.837 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:28:28.837 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:28:28.838 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:28:28.838 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:28:28.838 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:28:28.838 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:28:28.838 INFO:tasks.workunit.client.0.smithi046.stderr:303792: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:28:28.838 INFO:tasks.workunit.client.0.smithi046.stderr:303792: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:28:28.838 INFO:tasks.workunit.client.0.smithi046.stderr:303792: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:28:28.838 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:28:28.838 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:28:28.838 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:28:28.838 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:28:28.838 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:28:28.839 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:28:28.839 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:28:28.839 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:28:28.839 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:28:28.839 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:28:28.839 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:28:28.839 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-18T15:28:28.839 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-09-18T15:28:28.839 INFO:tasks.workunit.client.0.smithi046.stderr:303792: start osd.5 2024-09-18T15:28:29.106 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:28:29.106 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:28:29.106 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-09-18T15:28:29.106 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:28:29.106 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-18T15:28:29.106 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:28:29.106 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:28:29.106 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-09-18T15:28:29.106 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:28:29.106 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-18T15:28:29.106 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:28:29.106 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:28:29.107 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:28:29.107 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:28:29.107 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:28:29.107 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:28:29.107 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-18T15:28:29.107 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:28:29.107 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:28:29.107 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-09-18T15:28:29.107 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:28:29.107 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-18T15:28:29.107 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-18T15:28:29.107 INFO:tasks.workunit.client.0.smithi046.stderr:303795: _ 2024-09-18T15:28:29.107 INFO:tasks.workunit.client.0.smithi046.stderr:303795: hinfo_key 2024-09-18T15:28:29.107 INFO:tasks.workunit.client.0.smithi046.stderr:303795: snapset 2024-09-18T15:28:29.108 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-18T15:28:29.108 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:28:29.108 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:28:29.108 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-09-18T15:28:29.108 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:28:29.108 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-18T15:28:29.108 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:28:29.108 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:28:29.112 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:28:29.112 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:28:29.112 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-18T15:28:29.112 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-18T15:28:29.112 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:28:29.112 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:28:29.112 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:28:29.113 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:28:29.113 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:28:29.113 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:28:29.113 INFO:tasks.workunit.client.0.smithi046.stderr:303795: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:28:29.113 INFO:tasks.workunit.client.0.smithi046.stderr:303795: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:28:29.113 INFO:tasks.workunit.client.0.smithi046.stderr:303795: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:28:29.113 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:28:29.113 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:28:29.113 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:28:29.113 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:28:29.113 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:28:29.113 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:28:29.114 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:28:29.114 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:28:29.114 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:28:29.114 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:28:29.114 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:28:29.114 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-18T15:28:29.114 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-09-18T15:28:29.114 INFO:tasks.workunit.client.0.smithi046.stderr:303795: start osd.9 2024-09-18T15:28:33.665 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:28:33.666 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-18T15:28:33.666 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-09-18T15:28:33.666 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:28:33.666 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:28:33.666 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:28:33.666 INFO:tasks.workunit.client.0.smithi046.stderr:303792: 2024-09-18T15:28:28.856+0000 7f62a9d73540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:33.666 INFO:tasks.workunit.client.0.smithi046.stderr:303792: 2024-09-18T15:28:28.860+0000 7f62a9d73540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:33.666 INFO:tasks.workunit.client.0.smithi046.stderr:303792: 2024-09-18T15:28:28.860+0000 7f62a9d73540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:33.666 INFO:tasks.workunit.client.0.smithi046.stderr:303792: 2024-09-18T15:28:29.164+0000 7f62a9d73540 -1 Falling back to public interface 2024-09-18T15:28:33.666 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-09-18T15:28:33.666 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:28:33.667 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T15:28:33.667 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:28:33.667 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:28:33.667 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:33.667 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:28:33.667 INFO:tasks.workunit.client.0.smithi046.stderr:303792: 0 2024-09-18T15:28:33.667 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:33.667 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:28:33.667 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:33.667 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:33.667 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:33.667 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:28:33.668 INFO:tasks.workunit.client.0.smithi046.stderr:303792: 1 2024-09-18T15:28:33.668 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:33.668 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:28:33.668 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:33.668 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:33.668 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:33.668 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:28:33.668 INFO:tasks.workunit.client.0.smithi046.stderr:303792: 2 2024-09-18T15:28:33.668 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:33.668 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:28:33.668 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:33.668 INFO:tasks.workunit.client.0.smithi046.stderr:303792: 2024-09-18T15:28:32.288+0000 7f62a9d73540 -1 osd.5 100 log_to_monitors true 2024-09-18T15:28:33.669 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:33.669 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:33.669 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:28:33.669 INFO:tasks.workunit.client.0.smithi046.stderr:303792: 3 2024-09-18T15:28:33.669 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:33.669 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:28:33.902 INFO:tasks.workunit.client.0.smithi046.stderr:303792: osd.5 up in weight 1 up_from 104 up_thru 96 _recovery_ops 2024-09-18T15:28:33.902 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-18T15:28:33.902 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-09-18T15:28:33.902 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:28:33.902 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:28:33.902 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:28:33.902 INFO:tasks.workunit.client.0.smithi046.stderr:303795: 2024-09-18T15:28:29.128+0000 7fb01c2c2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:33.902 INFO:tasks.workunit.client.0.smithi046.stderr:303795: 2024-09-18T15:28:29.132+0000 7fb01c2c2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:33.902 INFO:tasks.workunit.client.0.smithi046.stderr:303795: 2024-09-18T15:28:29.136+0000 7fb01c2c2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:33.902 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-09-18T15:28:33.902 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:28:33.903 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-09-18T15:28:33.903 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:28:33.903 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:28:33.903 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:33.903 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:28:33.903 INFO:tasks.workunit.client.0.smithi046.stderr:303795: 0 2024-09-18T15:28:33.903 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:28:33.903 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:33.903 INFO:tasks.workunit.client.0.smithi046.stderr:303795: 2024-09-18T15:28:29.680+0000 7fb01c2c2540 -1 Falling back to public interface 2024-09-18T15:28:33.903 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:33.903 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:33.903 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:33.903 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:28:33.903 INFO:tasks.workunit.client.0.smithi046.stderr:303795: 1 2024-09-18T15:28:33.903 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:33.904 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:28:33.904 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:33.904 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:33.904 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:33.904 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:28:33.904 INFO:tasks.workunit.client.0.smithi046.stderr:303795: 2 2024-09-18T15:28:33.904 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:33.904 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:28:33.904 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:33.904 INFO:tasks.workunit.client.0.smithi046.stderr:303795: 2024-09-18T15:28:33.216+0000 7fb01c2c2540 -1 osd.9 100 log_to_monitors true 2024-09-18T15:28:33.904 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:33.904 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:33.904 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:28:33.904 INFO:tasks.workunit.client.0.smithi046.stderr:303795: 3 2024-09-18T15:28:33.905 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:33.905 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:28:34.679 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standaldown_at 101 last_clean_interval [96,100) [v2:127.0.0.1:6842/304599,v1:127.0.0.1:6843/304599] [v2:127.0.0.1:6844/304599,v1:127.0.0.1:6845/304599] exists,up 63244125-a488-402d-9bdf-d4f6f2b1fbf0 2024-09-18T15:28:34.679 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:28:34.679 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:28:34.679 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:28:34.679 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:28:34.680 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:28:34.680 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:28:34.680 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:28:34.680 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:28:34.680 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:28:34.680 INFO:tasks.workunit.client.0.smithi046.stderr:303792: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:28:34.680 INFO:tasks.workunit.client.0.smithi046.stderr:303792: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:28:34.680 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:28:34.680 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:28:34.680 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:28:34.680 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:28:34.681 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:28:34.681 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:28:34.681 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:28:34.681 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:28:34.681 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:28:34.681 INFO:tasks.workunit.client.0.smithi046.stderr:303792: 1 2024-09-18T15:28:34.681 INFO:tasks.workunit.client.0.smithi046.stderr:303792: 2 2024-09-18T15:28:34.681 INFO:tasks.workunit.client.0.smithi046.stderr:303792: 3 2024-09-18T15:28:34.681 INFO:tasks.workunit.client.0.smithi046.stderr:303792: 4 2024-09-18T15:28:34.681 INFO:tasks.workunit.client.0.smithi046.stderr:303792: 5 2024-09-18T15:28:34.682 INFO:tasks.workunit.client.0.smithi046.stderr:303792: 6 2024-09-18T15:28:34.682 INFO:tasks.workunit.client.0.smithi046.stderr:303792: 7 2024-09-18T15:28:34.682 INFO:tasks.workunit.client.0.smithi046.stderr:303792: 8 2024-09-18T15:28:34.682 INFO:tasks.workunit.client.0.smithi046.stderr:303792: 9' 2024-09-18T15:28:34.682 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:28:34.682 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:34.682 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:28:34.682 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836591 2024-09-18T15:28:34.682 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836591 2024-09-18T15:28:34.682 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836591' 2024-09-18T15:28:34.682 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:34.682 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:28:34.682 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673061 2024-09-18T15:28:34.682 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673061 2024-09-18T15:28:34.683 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836591 1-42949673061' 2024-09-18T15:28:34.683 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:36.444 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:36.444 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:36.444 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:36.444 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:28:36.444 INFO:tasks.workunit.client.0.smithi046.stderr:303795: 4 2024-09-18T15:28:36.444 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:36.445 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T15:28:36.445 INFO:tasks.workunit.client.0.smithi046.stderr:303795: osd.9 up in weight 1 up_from 106 up_thru 0 down_at 101 last_clean_interval [97,100) [v2:127.0.0.1:6874/304644,v1:127.0.0.1:6875/304644] [v2:127.0.0.1:6876/304644,v1:127.0.0.1:6877/304644] exists,up 364a066e-2eb8-45f2-a30f-be7da6149b02 2024-09-18T15:28:36.445 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:28:36.445 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:28:36.445 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:28:36.445 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:28:36.445 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:28:36.445 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:28:36.445 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:28:36.445 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:28:36.445 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:28:36.445 INFO:tasks.workunit.client.0.smithi046.stderr:303795: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:28:36.445 INFO:tasks.workunit.client.0.smithi046.stderr:303795: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:28:36.446 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:28:36.446 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:28:36.446 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:28:36.446 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:28:36.446 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:28:36.446 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:28:36.446 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:28:36.446 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:28:36.446 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:28:36.446 INFO:tasks.workunit.client.0.smithi046.stderr:303795: 1 2024-09-18T15:28:36.446 INFO:tasks.workunit.client.0.smithi046.stderr:303795: 2 2024-09-18T15:28:36.447 INFO:tasks.workunit.client.0.smithi046.stderr:303795: 3 2024-09-18T15:28:36.447 INFO:tasks.workunit.client.0.smithi046.stderr:303795: 4 2024-09-18T15:28:36.447 INFO:tasks.workunit.client.0.smithi046.stderr:303795: 5 2024-09-18T15:28:36.447 INFO:tasks.workunit.client.0.smithi046.stderr:303795: 6 2024-09-18T15:28:36.447 INFO:tasks.workunit.client.0.smithi046.stderr:303795: 7 2024-09-18T15:28:36.447 INFO:tasks.workunit.client.0.smithi046.stderr:303795: 8 2024-09-18T15:28:36.447 INFO:tasks.workunit.client.0.smithi046.stderr:303795: 9' 2024-09-18T15:28:36.447 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:28:36.447 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:36.447 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:28:36.447 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836592 2024-09-18T15:28:36.447 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836592 2024-09-18T15:28:36.447 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836592' 2024-09-18T15:28:36.447 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:36.732 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/cls.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:28:36.732 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509537 2024-09-18T15:28:36.732 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509537 2024-09-18T15:28:36.732 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836591 1-42949673061 2-64424509537' 2024-09-18T15:28:36.732 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:36.732 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:28:36.732 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=309237645342 2024-09-18T15:28:36.732 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 309237645342 2024-09-18T15:28:36.732 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836591 1-42949673061 2-64424509537 3-309237645342' 2024-09-18T15:28:36.732 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:36.733 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:28:36.733 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182482 2024-09-18T15:28:36.733 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182482 2024-09-18T15:28:36.733 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836591 1-42949673061 2-64424509537 3-309237645342 4-107374182482' 2024-09-18T15:28:36.733 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:36.733 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:28:36.733 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=446676598786 2024-09-18T15:28:36.733 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 446676598786 2024-09-18T15:28:36.733 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836591 1-42949673061 2-64424509537 3-309237645342 4-107374182482 5-446676598786' 2024-09-18T15:28:36.733 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:36.733 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:28:36.733 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855429 2024-09-18T15:28:36.733 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855429 2024-09-18T15:28:36.734 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836591 1-42949673061 2-64424509537 3-309237645342 4-107374182482 5-446676598786 6-150323855429' 2024-09-18T15:28:36.734 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:36.734 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:28:36.734 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691901 2024-09-18T15:28:36.734 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691901 2024-09-18T15:28:36.734 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836591 1-42949673061 2-64424509537 3-309237645342 4-107374182482 5-446676598786 6-150323855429 7-171798691901' 2024-09-18T15:28:36.734 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:36.734 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:28:37.466 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standaloneone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:28:37.466 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673062 2024-09-18T15:28:37.466 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673062 2024-09-18T15:28:37.466 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836592 1-42949673062' 2024-09-18T15:28:37.466 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:37.466 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:28:37.466 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509539 2024-09-18T15:28:37.466 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509539 2024-09-18T15:28:37.466 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836592 1-42949673062 2-64424509539' 2024-09-18T15:28:37.467 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:37.467 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:28:37.467 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=309237645343 2024-09-18T15:28:37.467 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 309237645343 2024-09-18T15:28:37.467 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836592 1-42949673062 2-64424509539 3-309237645343' 2024-09-18T15:28:37.467 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:37.467 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:28:37.467 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182483 2024-09-18T15:28:37.467 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182483 2024-09-18T15:28:37.467 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836592 1-42949673062 2-64424509539 3-309237645343 4-107374182483' 2024-09-18T15:28:37.467 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:37.467 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:28:37.467 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=446676598787 2024-09-18T15:28:37.468 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 446676598787 2024-09-18T15:28:37.468 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836592 1-42949673062 2-64424509539 3-309237645343 4-107374182483 5-446676598787' 2024-09-18T15:28:37.468 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:37.468 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:28:37.468 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855430 2024-09-18T15:28:37.468 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855430 2024-09-18T15:28:37.468 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836592 1-42949673062 2-64424509539 3-309237645343 4-107374182483 5-446676598787 6-150323855430' 2024-09-18T15:28:37.468 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:37.468 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:28:37.587 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528373 2024-09-18T15:28:37.588 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528373 2024-09-18T15:28:37.588 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836591 1-42949673061 2-64424509537 3-309237645342 4-107374182482 5-446676598786 6-150323855429 7-171798691901 8-193273528373' 2024-09-18T15:28:37.588 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:37.588 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:28:37.588 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=455266533378 2024-09-18T15:28:37.588 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 455266533378 2024-09-18T15:28:37.588 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836591 1-42949673061 2-64424509537 3-309237645342 4-107374182482 5-446676598786 6-150323855429 7-171798691901 8-193273528373 9-455266533378' 2024-09-18T15:28:37.588 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:37.588 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836591 2024-09-18T15:28:37.588 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:37.588 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:28:37.589 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:37.589 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836591 2024-09-18T15:28:37.589 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836591 2024-09-18T15:28:37.589 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836591' 2024-09-18T15:28:37.589 INFO:tasks.workunit.client.0.smithi046.stderr:303792: waiting osd.0 seq 21474836591 2024-09-18T15:28:37.589 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:28:37.589 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836592 -lt 21474836591 2024-09-18T15:28:37.589 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:37.589 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673061 2024-09-18T15:28:37.589 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:37.589 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:28:37.589 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673061 2024-09-18T15:28:37.589 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:37.589 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673061 2024-09-18T15:28:37.589 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673061' 2024-09-18T15:28:37.590 INFO:tasks.workunit.client.0.smithi046.stderr:303792: waiting osd.1 seq 42949673061 2024-09-18T15:28:37.590 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:28:37.590 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673062 -lt 42949673061 2024-09-18T15:28:37.590 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:37.590 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509537 2024-09-18T15:28:37.590 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:37.590 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:28:38.070 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/c902 2024-09-18T15:28:38.071 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691902 2024-09-18T15:28:38.071 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836592 1-42949673062 2-64424509539 3-309237645343 4-107374182483 5-446676598787 6-150323855430 7-171798691902' 2024-09-18T15:28:38.071 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:38.071 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:28:38.071 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528374 2024-09-18T15:28:38.071 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528374 2024-09-18T15:28:38.071 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836592 1-42949673062 2-64424509539 3-309237645343 4-107374182483 5-446676598787 6-150323855430 7-171798691902 8-193273528374' 2024-09-18T15:28:38.071 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:38.071 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:28:38.071 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=455266533379 2024-09-18T15:28:38.071 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 455266533379 2024-09-18T15:28:38.072 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836592 1-42949673062 2-64424509539 3-309237645343 4-107374182483 5-446676598787 6-150323855430 7-171798691902 8-193273528374 9-455266533379' 2024-09-18T15:28:38.072 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:38.072 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836592 2024-09-18T15:28:38.072 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:38.072 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:28:38.072 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836592 2024-09-18T15:28:38.072 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:38.072 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836592 2024-09-18T15:28:38.072 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836592' 2024-09-18T15:28:38.072 INFO:tasks.workunit.client.0.smithi046.stderr:303795: waiting osd.0 seq 21474836592 2024-09-18T15:28:38.072 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:28:38.072 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836592 -lt 21474836592 2024-09-18T15:28:38.072 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:38.072 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673062 2024-09-18T15:28:38.072 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:38.073 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:28:38.073 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673062 2024-09-18T15:28:38.073 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:38.073 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673062 2024-09-18T15:28:38.073 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673062' 2024-09-18T15:28:38.073 INFO:tasks.workunit.client.0.smithi046.stderr:303795: waiting osd.1 seq 42949673062 2024-09-18T15:28:38.619 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-staeph-helpers.sh:2183: flush_pg_stats: echo 2-64424509537 2024-09-18T15:28:38.620 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:38.620 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509537 2024-09-18T15:28:38.620 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509537' 2024-09-18T15:28:38.620 INFO:tasks.workunit.client.0.smithi046.stderr:303792: waiting osd.2 seq 64424509537 2024-09-18T15:28:38.620 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:28:38.620 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509539 -lt 64424509537 2024-09-18T15:28:38.620 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:38.620 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:38.620 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-309237645342 2024-09-18T15:28:38.620 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:28:38.620 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-309237645342 2024-09-18T15:28:38.621 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:38.621 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=309237645342 2024-09-18T15:28:38.621 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 309237645342' 2024-09-18T15:28:38.621 INFO:tasks.workunit.client.0.smithi046.stderr:303792: waiting osd.3 seq 309237645342 2024-09-18T15:28:38.621 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:28:38.621 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 309237645343 -lt 309237645342 2024-09-18T15:28:38.621 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:38.621 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182482 2024-09-18T15:28:38.621 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:38.621 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:28:38.621 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:38.621 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182482 2024-09-18T15:28:38.621 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182482 2024-09-18T15:28:38.621 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182482' 2024-09-18T15:28:38.622 INFO:tasks.workunit.client.0.smithi046.stderr:303792: waiting osd.4 seq 107374182482 2024-09-18T15:28:38.622 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:28:38.622 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182482 -lt 107374182482 2024-09-18T15:28:38.622 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:38.622 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-446676598786 2024-09-18T15:28:38.622 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:38.622 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:28:38.622 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-446676598786 2024-09-18T15:28:38.622 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:38.622 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=446676598786 2024-09-18T15:28:39.452 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 4466765987t-seq 1 2024-09-18T15:28:39.453 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673062 -lt 42949673062 2024-09-18T15:28:39.453 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:39.453 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509539 2024-09-18T15:28:39.453 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:39.453 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:28:39.453 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509539 2024-09-18T15:28:39.453 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:39.453 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509539 2024-09-18T15:28:39.453 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509539' 2024-09-18T15:28:39.453 INFO:tasks.workunit.client.0.smithi046.stderr:303795: waiting osd.2 seq 64424509539 2024-09-18T15:28:39.454 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:28:39.454 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509539 -lt 64424509539 2024-09-18T15:28:39.454 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:39.454 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-309237645343 2024-09-18T15:28:39.454 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:39.454 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:28:39.454 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-309237645343 2024-09-18T15:28:39.454 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:39.454 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=309237645343 2024-09-18T15:28:39.454 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 309237645343' 2024-09-18T15:28:39.454 INFO:tasks.workunit.client.0.smithi046.stderr:303795: waiting osd.3 seq 309237645343 2024-09-18T15:28:39.454 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:28:39.454 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 309237645343 -lt 309237645343 2024-09-18T15:28:39.455 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:39.455 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182483 2024-09-18T15:28:39.455 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:39.455 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:28:39.455 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182483 2024-09-18T15:28:39.455 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:39.455 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182483 2024-09-18T15:28:39.455 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182483' 2024-09-18T15:28:39.455 INFO:tasks.workunit.client.0.smithi046.stderr:303795: waiting osd.4 seq 107374182483 2024-09-18T15:28:39.455 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:28:39.455 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182484 -lt 107374182483 2024-09-18T15:28:39.455 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:39.455 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-446676598787 2024-09-18T15:28:39.994 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa86' 2024-09-18T15:28:39.994 INFO:tasks.workunit.client.0.smithi046.stderr:303792: waiting osd.5 seq 446676598786 2024-09-18T15:28:39.995 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:28:39.995 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 446676598786 -lt 446676598786 2024-09-18T15:28:39.995 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:39.995 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855429 2024-09-18T15:28:39.995 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:39.995 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:28:39.995 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855429 2024-09-18T15:28:39.995 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:39.995 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855429 2024-09-18T15:28:39.995 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855429' 2024-09-18T15:28:39.995 INFO:tasks.workunit.client.0.smithi046.stderr:303792: waiting osd.6 seq 150323855429 2024-09-18T15:28:39.995 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:28:39.996 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855431 -lt 150323855429 2024-09-18T15:28:39.996 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:39.996 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691901 2024-09-18T15:28:39.996 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:39.996 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:28:39.996 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691901 2024-09-18T15:28:39.996 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:39.996 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691901 2024-09-18T15:28:39.996 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691901' 2024-09-18T15:28:39.996 INFO:tasks.workunit.client.0.smithi046.stderr:303792: waiting osd.7 seq 171798691901 2024-09-18T15:28:39.996 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:28:39.996 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691902 -lt 171798691901 2024-09-18T15:28:39.997 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:39.997 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528373 2024-09-18T15:28:39.997 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:39.997 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:28:39.997 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528373 2024-09-18T15:28:39.997 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:39.997 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528373 2024-09-18T15:28:39.997 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528373' 2024-09-18T15:28:39.997 INFO:tasks.workunit.client.0.smithi046.stderr:303792: waiting osd.8 seq 193273528373 2024-09-18T15:28:39.997 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:28:39.997 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528374 -lt 193273528373 2024-09-18T15:28:40.523 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:40.523 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:28:40.523 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-446676598787 2024-09-18T15:28:40.523 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:40.523 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=446676598787 2024-09-18T15:28:40.523 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 446676598787' 2024-09-18T15:28:40.523 INFO:tasks.workunit.client.0.smithi046.stderr:303795: waiting osd.5 seq 446676598787 2024-09-18T15:28:40.523 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:28:40.523 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 446676598788 -lt 446676598787 2024-09-18T15:28:40.524 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:40.524 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855430 2024-09-18T15:28:40.524 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:40.524 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:28:40.524 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855430 2024-09-18T15:28:40.524 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:40.524 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855430 2024-09-18T15:28:40.524 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855430' 2024-09-18T15:28:40.524 INFO:tasks.workunit.client.0.smithi046.stderr:303795: waiting osd.6 seq 150323855430 2024-09-18T15:28:40.524 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:28:40.524 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855431 -lt 150323855430 2024-09-18T15:28:40.524 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:40.524 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:40.524 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691902 2024-09-18T15:28:40.525 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:28:40.525 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691902 2024-09-18T15:28:40.525 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:40.525 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691902 2024-09-18T15:28:40.525 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691902' 2024-09-18T15:28:40.525 INFO:tasks.workunit.client.0.smithi046.stderr:303795: waiting osd.7 seq 171798691902 2024-09-18T15:28:40.525 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:28:40.525 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691902 -lt 171798691902 2024-09-18T15:28:40.525 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:40.525 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528374 2024-09-18T15:28:40.525 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:40.525 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:28:40.525 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528374 2024-09-18T15:28:40.526 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:41.449 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /ho 2024-09-18T15:28:41.449 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-455266533378 2024-09-18T15:28:41.449 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:41.449 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:28:41.449 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-455266533378 2024-09-18T15:28:41.449 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:41.449 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=455266533378 2024-09-18T15:28:41.449 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 455266533378' 2024-09-18T15:28:41.449 INFO:tasks.workunit.client.0.smithi046.stderr:303792: waiting osd.9 seq 455266533378 2024-09-18T15:28:41.449 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:28:41.449 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 455266533379 -lt 455266533378 2024-09-18T15:28:41.449 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:28:41.450 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:28:41.450 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:28:41.450 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:28:41.450 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:28:41.450 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:28:41.450 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:28:41.450 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:28:41.450 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:28:41.450 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:28:41.450 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:28:41.450 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:28:41.450 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:28:41.450 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:28:41.450 INFO:tasks.workunit.client.0.smithi046.stderr:303792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:28:41.451 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:28:41.451 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:28:41.451 INFO:tasks.workunit.client.0.smithi046.stderr:303792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:28:41.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:28:41.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:28:41.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 303790 2024-09-18T15:28:42.347 INFO:tasks.workunit.client.0.smithi046.stderr:me/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528374 2024-09-18T15:28:42.347 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528374' 2024-09-18T15:28:42.348 INFO:tasks.workunit.client.0.smithi046.stderr:303795: waiting osd.8 seq 193273528374 2024-09-18T15:28:42.348 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:28:42.348 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528374 -lt 193273528374 2024-09-18T15:28:42.348 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:42.348 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-455266533379 2024-09-18T15:28:42.348 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:42.348 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:28:42.348 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-455266533379 2024-09-18T15:28:42.348 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:42.348 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=455266533379 2024-09-18T15:28:42.348 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 455266533379' 2024-09-18T15:28:42.348 INFO:tasks.workunit.client.0.smithi046.stderr:303795: waiting osd.9 seq 455266533379 2024-09-18T15:28:42.348 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:28:42.349 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 455266533380 -lt 455266533379 2024-09-18T15:28:42.349 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:28:42.349 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:28:42.349 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:28:42.349 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:28:42.349 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:28:42.349 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:28:42.349 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:28:42.349 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:28:42.349 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:28:42.349 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:28:42.349 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:28:42.349 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:28:42.350 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:28:42.350 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:28:42.350 INFO:tasks.workunit.client.0.smithi046.stderr:303795: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:28:42.350 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:28:42.350 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:28:42.350 INFO:tasks.workunit.client.0.smithi046.stderr:303795: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:28:42.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:28:42.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: eval 'pids='\'''\''' 2024-09-18T15:28:42.350 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: pids= 2024-09-18T15:28:42.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: wait_background: return 0 2024-09-18T15:28:42.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: return_code=0 2024-09-18T15:28:42.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-18T15:28:42.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:214: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-18T15:28:42.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-18T15:28:42.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-09-18T15:28:42.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:184: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-18T15:28:42.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local poolname=ecpool 2024-09-18T15:28:42.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local first=3 2024-09-18T15:28:42.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local second=5 2024-09-18T15:28:42.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:192: corrupt_and_repair_two: pids= 2024-09-18T15:28:42.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:28:42.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:28:42.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:28:42.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 307229"' 2024-09-18T15:28:42.401 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 307229' 2024-09-18T15:28:42.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-18T15:28:42.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:28:42.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:28:42.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 307230"' 2024-09-18T15:28:42.401 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 307230' 2024-09-18T15:28:42.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: wait_background pids 2024-09-18T15:28:42.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: wait_background: pids=' 307229 307230' 2024-09-18T15:28:42.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: wait_background: return_code=0 2024-09-18T15:28:42.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:28:42.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 307229 2024-09-18T15:28:42.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:28:42.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/307232: /' 2024-09-18T15:28:42.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-18T15:28:42.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/307234: /' 2024-09-18T15:28:46.243 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:28:46.243 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:28:46.243 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-18T15:28:46.243 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:28:46.243 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:28:46.243 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:28:46.243 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:28:46.244 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-18T15:28:46.244 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:28:46.244 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-18T15:28:46.244 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:28:46.244 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:28:46.244 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:28:46.244 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:28:46.244 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:28:46.244 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:28:46.244 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T15:28:46.244 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:28:46.244 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:28:46.244 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-18T15:28:46.244 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:28:46.245 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:28:46.245 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-18T15:28:46.245 INFO:tasks.workunit.client.0.smithi046.stderr:307232: remove 0#2:eb822e21:::SOMETHING:head# 2024-09-18T15:28:46.245 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-18T15:28:46.245 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:28:46.245 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:28:46.245 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T15:28:46.245 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:28:46.245 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:28:46.245 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:28:46.245 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:28:46.250 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:28:46.250 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:28:46.250 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-18T15:28:46.250 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-18T15:28:46.250 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:28:46.250 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:28:46.250 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:28:46.250 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:28:46.250 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:28:46.250 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:28:46.251 INFO:tasks.workunit.client.0.smithi046.stderr:307232: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:28:46.251 INFO:tasks.workunit.client.0.smithi046.stderr:307232: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:28:46.251 INFO:tasks.workunit.client.0.smithi046.stderr:307232: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:28:46.251 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:28:46.251 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:28:46.251 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:28:46.251 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:28:46.251 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:28:46.251 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:28:46.251 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:28:46.251 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:28:46.251 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:28:46.251 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:28:46.251 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:28:46.252 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-18T15:28:46.252 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T15:28:46.252 INFO:tasks.workunit.client.0.smithi046.stderr:307232: start osd.3 2024-09-18T15:28:46.414 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:28:46.414 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:28:46.414 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-09-18T15:28:46.414 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:28:46.414 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-18T15:28:46.415 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:28:46.415 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:28:46.415 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-09-18T15:28:46.415 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:28:46.415 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-18T15:28:46.415 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:28:46.415 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:28:46.415 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:28:46.415 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:28:46.415 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:28:46.415 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:28:46.415 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-18T15:28:46.415 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:28:46.416 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:28:46.416 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-09-18T15:28:46.416 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:28:46.416 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-18T15:28:46.416 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-18T15:28:46.416 INFO:tasks.workunit.client.0.smithi046.stderr:307234: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-18T15:28:46.416 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-18T15:28:46.416 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:28:46.416 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:28:46.416 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-09-18T15:28:46.416 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:28:46.416 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-18T15:28:46.416 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:28:46.416 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:28:46.420 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:28:46.420 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:28:46.420 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-18T15:28:46.420 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-18T15:28:46.420 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:28:46.420 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:28:46.421 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:28:46.421 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:28:46.421 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:28:46.421 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:28:46.421 INFO:tasks.workunit.client.0.smithi046.stderr:307234: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:28:46.421 INFO:tasks.workunit.client.0.smithi046.stderr:307234: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:28:46.421 INFO:tasks.workunit.client.0.smithi046.stderr:307234: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:28:46.421 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:28:46.421 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:28:46.421 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:28:46.421 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:28:46.421 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:28:46.422 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:28:46.422 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:28:46.422 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:28:46.422 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:28:46.422 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:28:46.422 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:28:46.422 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-18T15:28:46.422 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-09-18T15:28:46.422 INFO:tasks.workunit.client.0.smithi046.stderr:307234: start osd.5 2024-09-18T15:28:51.403 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:28:51.403 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-18T15:28:51.403 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T15:28:51.403 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:28:51.403 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:28:51.403 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:28:51.403 INFO:tasks.workunit.client.0.smithi046.stderr:307232: 2024-09-18T15:28:46.276+0000 7f7273ca4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:51.404 INFO:tasks.workunit.client.0.smithi046.stderr:307232: 2024-09-18T15:28:46.280+0000 7f7273ca4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:51.404 INFO:tasks.workunit.client.0.smithi046.stderr:307232: 2024-09-18T15:28:46.280+0000 7f7273ca4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:51.404 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T15:28:51.404 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:28:51.404 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:28:51.404 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:28:51.404 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:28:51.404 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:51.404 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:28:51.404 INFO:tasks.workunit.client.0.smithi046.stderr:307232: 0 2024-09-18T15:28:51.405 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:51.405 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:28:51.405 INFO:tasks.workunit.client.0.smithi046.stderr:307232: 2024-09-18T15:28:46.828+0000 7f7273ca4540 -1 Falling back to public interface 2024-09-18T15:28:51.405 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:51.405 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:51.405 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:51.405 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:28:51.405 INFO:tasks.workunit.client.0.smithi046.stderr:307232: 1 2024-09-18T15:28:51.405 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:51.405 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:28:51.405 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:51.405 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:51.405 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:51.405 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:28:51.406 INFO:tasks.workunit.client.0.smithi046.stderr:307232: 2 2024-09-18T15:28:51.406 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:51.406 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:28:51.406 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:51.406 INFO:tasks.workunit.client.0.smithi046.stderr:307232: 2024-09-18T15:28:49.948+0000 7f7273ca4540 -1 osd.3 108 log_to_monitors true 2024-09-18T15:28:51.406 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:51.406 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:51.406 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:28:51.406 INFO:tasks.workunit.client.0.smithi046.stderr:307232: 3 2024-09-18T15:28:51.406 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:51.406 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:28:51.663 INFO:tasks.workunit.client.0.smithi046.stderr:307232: osd.3 up in weight 1 up_from 113 up_thru 106 down_at 1ops 2024-09-18T15:28:51.663 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-18T15:28:51.663 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-09-18T15:28:51.663 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:28:51.664 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:28:51.664 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:28:51.664 INFO:tasks.workunit.client.0.smithi046.stderr:307234: 2024-09-18T15:28:46.440+0000 7f02ab98a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:51.664 INFO:tasks.workunit.client.0.smithi046.stderr:307234: 2024-09-18T15:28:46.444+0000 7f02ab98a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:51.664 INFO:tasks.workunit.client.0.smithi046.stderr:307234: 2024-09-18T15:28:46.444+0000 7f02ab98a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:51.664 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-09-18T15:28:51.664 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:28:51.664 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T15:28:51.664 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:28:51.664 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:28:51.664 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:51.665 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:28:51.665 INFO:tasks.workunit.client.0.smithi046.stderr:307234: 0 2024-09-18T15:28:51.665 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:51.665 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:28:51.665 INFO:tasks.workunit.client.0.smithi046.stderr:307234: 2024-09-18T15:28:46.996+0000 7f02ab98a540 -1 Falling back to public interface 2024-09-18T15:28:51.665 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:51.665 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:51.665 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:51.665 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:28:51.665 INFO:tasks.workunit.client.0.smithi046.stderr:307234: 1 2024-09-18T15:28:51.665 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:51.665 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:28:51.665 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:51.665 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:51.666 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:51.666 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:28:51.666 INFO:tasks.workunit.client.0.smithi046.stderr:307234: 2 2024-09-18T15:28:51.666 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:51.666 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:28:51.666 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:51.666 INFO:tasks.workunit.client.0.smithi046.stderr:307234: 2024-09-18T15:28:50.540+0000 7f02ab98a540 -1 osd.5 108 log_to_monitors true 2024-09-18T15:28:51.666 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:51.666 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:51.666 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:28:51.666 INFO:tasks.workunit.client.0.smithi046.stderr:307234: 3 2024-09-18T15:28:51.666 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:51.666 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:28:52.727 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-h09 last_clean_interval [72,108) [v2:127.0.0.1:6826/308039,v1:127.0.0.1:6827/308039] [v2:127.0.0.1:6828/308039,v1:127.0.0.1:6829/308039] exists,up 105fcb65-1d2b-4358-9b52-7d1b65a95f69 2024-09-18T15:28:52.727 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:28:52.727 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:28:52.727 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:28:52.727 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:28:52.727 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:28:52.727 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:28:52.727 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:28:52.727 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:28:52.727 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:28:52.727 INFO:tasks.workunit.client.0.smithi046.stderr:307232: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:28:52.727 INFO:tasks.workunit.client.0.smithi046.stderr:307232: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:28:52.728 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:28:52.728 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:28:52.728 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:28:52.728 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:28:52.728 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:28:52.728 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:28:52.728 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:28:52.728 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:28:52.728 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:28:52.728 INFO:tasks.workunit.client.0.smithi046.stderr:307232: 1 2024-09-18T15:28:52.728 INFO:tasks.workunit.client.0.smithi046.stderr:307232: 2 2024-09-18T15:28:52.728 INFO:tasks.workunit.client.0.smithi046.stderr:307232: 3 2024-09-18T15:28:52.728 INFO:tasks.workunit.client.0.smithi046.stderr:307232: 4 2024-09-18T15:28:52.729 INFO:tasks.workunit.client.0.smithi046.stderr:307232: 5 2024-09-18T15:28:52.729 INFO:tasks.workunit.client.0.smithi046.stderr:307232: 6 2024-09-18T15:28:52.729 INFO:tasks.workunit.client.0.smithi046.stderr:307232: 7 2024-09-18T15:28:52.729 INFO:tasks.workunit.client.0.smithi046.stderr:307232: 8 2024-09-18T15:28:52.729 INFO:tasks.workunit.client.0.smithi046.stderr:307232: 9' 2024-09-18T15:28:52.729 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:28:52.729 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:52.729 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:28:52.729 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836596 2024-09-18T15:28:52.729 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836596 2024-09-18T15:28:52.729 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836596' 2024-09-18T15:28:52.729 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:52.729 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:28:52.729 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673066 2024-09-18T15:28:52.729 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673066 2024-09-18T15:28:52.730 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836596 1-42949673066' 2024-09-18T15:28:52.730 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:54.040 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:28:54.040 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509543 2024-09-18T15:28:54.041 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509543 2024-09-18T15:28:54.041 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836596 1-42949673066 2-64424509543' 2024-09-18T15:28:54.041 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:54.041 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:28:54.041 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=485331304450 2024-09-18T15:28:54.041 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 485331304450 2024-09-18T15:28:54.041 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836596 1-42949673066 2-64424509543 3-485331304450' 2024-09-18T15:28:54.041 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:54.041 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:28:54.041 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182488 2024-09-18T15:28:54.041 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182488 2024-09-18T15:28:54.041 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836596 1-42949673066 2-64424509543 3-485331304450 4-107374182488' 2024-09-18T15:28:54.041 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:54.042 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:28:54.042 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=498216206338 2024-09-18T15:28:54.042 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 498216206338 2024-09-18T15:28:54.042 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836596 1-42949673066 2-64424509543 3-485331304450 4-107374182488 5-498216206338' 2024-09-18T15:28:54.042 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:54.042 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:28:54.042 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855435 2024-09-18T15:28:54.042 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855435 2024-09-18T15:28:54.042 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836596 1-42949673066 2-64424509543 3-485331304450 4-107374182488 5-498216206338 6-150323855435' 2024-09-18T15:28:54.042 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:54.042 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:28:54.042 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691906 2024-09-18T15:28:54.042 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691906 2024-09-18T15:28:54.043 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836596 1-42949673066 2-64424509543 3-485331304450 4-107374182488 5-498216206338 6-150323855435 7-171798691906' 2024-09-18T15:28:54.043 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:54.043 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:28:54.307 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helelpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:54.307 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:54.307 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:54.307 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:28:54.307 INFO:tasks.workunit.client.0.smithi046.stderr:307234: 4 2024-09-18T15:28:54.307 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:54.307 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:28:54.307 INFO:tasks.workunit.client.0.smithi046.stderr:307234: osd.5 up in weight 1 up_from 116 up_thru 104 down_at 109 last_clean_interval [104,108) [v2:127.0.0.1:6842/308084,v1:127.0.0.1:6843/308084] [v2:127.0.0.1:6844/308084,v1:127.0.0.1:6845/308084] exists,up 63244125-a488-402d-9bdf-d4f6f2b1fbf0 2024-09-18T15:28:54.307 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:28:54.307 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:28:54.307 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:28:54.307 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:28:54.308 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:28:54.308 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:28:54.308 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:28:54.308 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:28:54.308 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:28:54.308 INFO:tasks.workunit.client.0.smithi046.stderr:307234: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:28:54.308 INFO:tasks.workunit.client.0.smithi046.stderr:307234: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:28:54.308 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:28:54.308 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:28:54.308 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:28:54.308 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:28:54.308 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:28:54.309 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:28:54.309 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:28:54.309 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:28:54.309 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:28:54.309 INFO:tasks.workunit.client.0.smithi046.stderr:307234: 1 2024-09-18T15:28:54.309 INFO:tasks.workunit.client.0.smithi046.stderr:307234: 2 2024-09-18T15:28:54.309 INFO:tasks.workunit.client.0.smithi046.stderr:307234: 3 2024-09-18T15:28:54.309 INFO:tasks.workunit.client.0.smithi046.stderr:307234: 4 2024-09-18T15:28:54.309 INFO:tasks.workunit.client.0.smithi046.stderr:307234: 5 2024-09-18T15:28:54.309 INFO:tasks.workunit.client.0.smithi046.stderr:307234: 6 2024-09-18T15:28:54.309 INFO:tasks.workunit.client.0.smithi046.stderr:307234: 7 2024-09-18T15:28:54.309 INFO:tasks.workunit.client.0.smithi046.stderr:307234: 8 2024-09-18T15:28:54.309 INFO:tasks.workunit.client.0.smithi046.stderr:307234: 9' 2024-09-18T15:28:54.309 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:28:54.309 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:54.310 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:28:54.310 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836598 2024-09-18T15:28:54.310 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836598 2024-09-18T15:28:54.310 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598' 2024-09-18T15:28:54.310 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:54.988 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.clipers.sh:2173: flush_pg_stats: seq=193273528378 2024-09-18T15:28:54.988 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528378 2024-09-18T15:28:54.988 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836596 1-42949673066 2-64424509543 3-485331304450 4-107374182488 5-498216206338 6-150323855435 7-171798691906 8-193273528378' 2024-09-18T15:28:54.988 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:54.988 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:28:54.988 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=455266533384 2024-09-18T15:28:54.988 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 455266533384 2024-09-18T15:28:54.988 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836596 1-42949673066 2-64424509543 3-485331304450 4-107374182488 5-498216206338 6-150323855435 7-171798691906 8-193273528378 9-455266533384' 2024-09-18T15:28:54.988 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:54.988 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836596 2024-09-18T15:28:54.988 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:54.989 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:28:54.989 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836596 2024-09-18T15:28:54.989 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:54.989 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836596 2024-09-18T15:28:54.989 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836596' 2024-09-18T15:28:54.989 INFO:tasks.workunit.client.0.smithi046.stderr:307232: waiting osd.0 seq 21474836596 2024-09-18T15:28:54.989 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:28:54.989 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836596 -lt 21474836596 2024-09-18T15:28:54.989 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:54.989 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673066 2024-09-18T15:28:54.989 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:54.989 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:28:54.989 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673066 2024-09-18T15:28:54.990 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:54.990 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673066 2024-09-18T15:28:54.990 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673066' 2024-09-18T15:28:54.990 INFO:tasks.workunit.client.0.smithi046.stderr:307232: waiting osd.1 seq 42949673066 2024-09-18T15:28:54.990 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:28:54.990 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673066 -lt 42949673066 2024-09-18T15:28:54.990 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:54.990 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509543 2024-09-18T15:28:54.990 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:54.990 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:28:55.248 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeent.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:28:55.248 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673068 2024-09-18T15:28:55.248 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673068 2024-09-18T15:28:55.248 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673068' 2024-09-18T15:28:55.248 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:55.248 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:28:55.248 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509544 2024-09-18T15:28:55.249 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509544 2024-09-18T15:28:55.249 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673068 2-64424509544' 2024-09-18T15:28:55.249 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:55.249 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:28:55.249 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=485331304451 2024-09-18T15:28:55.249 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 485331304451 2024-09-18T15:28:55.249 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673068 2-64424509544 3-485331304451' 2024-09-18T15:28:55.249 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:55.249 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:28:55.249 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182489 2024-09-18T15:28:55.249 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182489 2024-09-18T15:28:55.250 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673068 2-64424509544 3-485331304451 4-107374182489' 2024-09-18T15:28:55.250 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:55.250 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:28:55.250 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=498216206339 2024-09-18T15:28:55.250 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 498216206339 2024-09-18T15:28:55.250 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673068 2-64424509544 3-485331304451 4-107374182489 5-498216206339' 2024-09-18T15:28:55.250 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:55.250 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:28:55.250 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855436 2024-09-18T15:28:55.250 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855436 2024-09-18T15:28:55.250 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673068 2-64424509544 3-485331304451 4-107374182489 5-498216206339 6-150323855436' 2024-09-18T15:28:55.250 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:55.250 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:28:55.251 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691908 2024-09-18T15:28:55.820 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691908 2024-09-18T15:28:55.820 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673068 2-64424509544 3-485331304451 4-107374182489 5-498216206339 6-150323855436 7-171798691908' 2024-09-18T15:28:55.820 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:55.820 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:28:55.820 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528379 2024-09-18T15:28:55.820 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528379 2024-09-18T15:28:55.820 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673068 2-64424509544 3-485331304451 4-107374182489 5-498216206339 6-150323855436 7-171798691908 8-193273528379' 2024-09-18T15:28:55.821 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:55.821 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:28:55.821 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=455266533385 2024-09-18T15:28:55.821 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 455266533385 2024-09-18T15:28:55.821 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673068 2-64424509544 3-485331304451 4-107374182489 5-498216206339 6-150323855436 7-171798691908 8-193273528379 9-455266533385' 2024-09-18T15:28:55.821 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:55.821 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836598 2024-09-18T15:28:55.821 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:55.821 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:28:55.821 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836598 2024-09-18T15:28:55.821 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:55.821 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836598 2024-09-18T15:28:55.821 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836598' 2024-09-18T15:28:55.821 INFO:tasks.workunit.client.0.smithi046.stderr:307234: waiting osd.0 seq 21474836598 2024-09-18T15:28:55.822 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:28:55.822 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836598 -lt 21474836598 2024-09-18T15:28:55.822 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:55.822 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673068 2024-09-18T15:28:55.822 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:55.822 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:28:55.822 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673068 2024-09-18T15:28:55.822 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:55.822 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673068 2024-09-18T15:28:55.822 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673068' 2024-09-18T15:28:55.822 INFO:tasks.workunit.client.0.smithi046.stderr:307234: waiting osd.1 seq 42949673068 2024-09-18T15:28:55.986 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1rs.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:55.986 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509543 2024-09-18T15:28:55.986 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509543 2024-09-18T15:28:55.986 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509543' 2024-09-18T15:28:55.986 INFO:tasks.workunit.client.0.smithi046.stderr:307232: waiting osd.2 seq 64424509543 2024-09-18T15:28:55.987 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:28:55.987 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509544 -lt 64424509543 2024-09-18T15:28:55.987 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:55.987 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-485331304450 2024-09-18T15:28:55.987 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:55.987 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:28:55.987 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-485331304450 2024-09-18T15:28:55.987 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:55.987 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=485331304450 2024-09-18T15:28:55.987 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 485331304450' 2024-09-18T15:28:55.987 INFO:tasks.workunit.client.0.smithi046.stderr:307232: waiting osd.3 seq 485331304450 2024-09-18T15:28:55.987 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:28:55.987 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 485331304451 -lt 485331304450 2024-09-18T15:28:55.987 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:55.988 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182488 2024-09-18T15:28:55.988 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:55.988 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:28:55.988 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182488 2024-09-18T15:28:55.988 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:55.988 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182488 2024-09-18T15:28:55.988 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182488' 2024-09-18T15:28:55.988 INFO:tasks.workunit.client.0.smithi046.stderr:307232: waiting osd.4 seq 107374182488 2024-09-18T15:28:55.988 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:28:55.988 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182489 -lt 107374182488 2024-09-18T15:28:55.988 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:55.988 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-498216206338 2024-09-18T15:28:55.988 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:55.988 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:28:55.988 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-498216206338 2024-09-18T15:28:55.989 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:55.989 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=498216206338 2024-09-18T15:28:55.989 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 498216206338' 2024-09-18T15:28:57.190 INFO:tasks.workunit.client.0.smithi046.stderr:30723 2024-09-18T15:28:57.190 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673068 -lt 42949673068 2024-09-18T15:28:57.190 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:57.190 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509544 2024-09-18T15:28:57.190 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:57.190 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:28:57.190 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509544 2024-09-18T15:28:57.190 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:57.191 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509544 2024-09-18T15:28:57.191 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509544' 2024-09-18T15:28:57.191 INFO:tasks.workunit.client.0.smithi046.stderr:307234: waiting osd.2 seq 64424509544 2024-09-18T15:28:57.191 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:28:57.191 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509544 -lt 64424509544 2024-09-18T15:28:57.191 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:57.191 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-485331304451 2024-09-18T15:28:57.191 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:57.191 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:28:57.191 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-485331304451 2024-09-18T15:28:57.191 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:57.191 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=485331304451 2024-09-18T15:28:57.192 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 485331304451' 2024-09-18T15:28:57.192 INFO:tasks.workunit.client.0.smithi046.stderr:307234: waiting osd.3 seq 485331304451 2024-09-18T15:28:57.192 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:28:57.192 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 485331304451 -lt 485331304451 2024-09-18T15:28:57.192 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:57.192 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182489 2024-09-18T15:28:57.192 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:57.192 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:28:57.192 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182489 2024-09-18T15:28:57.192 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:57.192 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182489 2024-09-18T15:28:57.192 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182489' 2024-09-18T15:28:57.192 INFO:tasks.workunit.client.0.smithi046.stderr:307234: waiting osd.4 seq 107374182489 2024-09-18T15:28:57.193 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:28:57.193 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182489 -lt 107374182489 2024-09-18T15:28:57.193 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:57.193 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-498216206339 2024-09-18T15:28:57.341 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standa2: waiting osd.5 seq 498216206338 2024-09-18T15:28:57.341 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:28:57.341 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 498216206338 -lt 498216206338 2024-09-18T15:28:57.341 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:57.341 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855435 2024-09-18T15:28:57.341 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:57.341 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:28:57.341 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:57.341 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855435 2024-09-18T15:28:57.342 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855435 2024-09-18T15:28:57.342 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855435' 2024-09-18T15:28:57.342 INFO:tasks.workunit.client.0.smithi046.stderr:307232: waiting osd.6 seq 150323855435 2024-09-18T15:28:57.342 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:28:57.342 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855435 -lt 150323855435 2024-09-18T15:28:57.342 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:57.342 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:57.342 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691906 2024-09-18T15:28:57.342 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:28:57.342 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691906 2024-09-18T15:28:57.342 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:57.342 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691906 2024-09-18T15:28:57.342 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691906' 2024-09-18T15:28:57.343 INFO:tasks.workunit.client.0.smithi046.stderr:307232: waiting osd.7 seq 171798691906 2024-09-18T15:28:57.343 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:28:57.343 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691907 -lt 171798691906 2024-09-18T15:28:57.343 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:57.343 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:57.343 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528378 2024-09-18T15:28:57.343 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:28:57.343 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528378 2024-09-18T15:28:57.343 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:57.343 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528378 2024-09-18T15:28:57.343 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528378' 2024-09-18T15:28:57.343 INFO:tasks.workunit.client.0.smithi046.stderr:307232: waiting osd.8 seq 193273528378 2024-09-18T15:28:57.343 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:28:57.343 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528380 -lt 193273528378 2024-09-18T15:28:57.344 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:58.193 INFO:tasks.workunit.client.0.smithi046.stderr:307232: lone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:58.194 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:28:58.194 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-498216206339 2024-09-18T15:28:58.194 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:58.194 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=498216206339 2024-09-18T15:28:58.194 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 498216206339' 2024-09-18T15:28:58.194 INFO:tasks.workunit.client.0.smithi046.stderr:307234: waiting osd.5 seq 498216206339 2024-09-18T15:28:58.194 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:28:58.194 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 498216206339 -lt 498216206339 2024-09-18T15:28:58.194 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:58.194 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855436 2024-09-18T15:28:58.194 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:58.194 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:28:58.195 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855436 2024-09-18T15:28:58.195 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:58.195 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855436 2024-09-18T15:28:58.195 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855436' 2024-09-18T15:28:58.195 INFO:tasks.workunit.client.0.smithi046.stderr:307234: waiting osd.6 seq 150323855436 2024-09-18T15:28:58.195 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:28:58.195 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855436 -lt 150323855436 2024-09-18T15:28:58.195 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:58.195 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691908 2024-09-18T15:28:58.195 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:58.195 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:28:58.195 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691908 2024-09-18T15:28:58.195 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:58.196 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691908 2024-09-18T15:28:58.196 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691908' 2024-09-18T15:28:58.196 INFO:tasks.workunit.client.0.smithi046.stderr:307234: waiting osd.7 seq 171798691908 2024-09-18T15:28:58.196 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:28:58.196 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691908 -lt 171798691908 2024-09-18T15:28:58.196 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:58.196 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528379 2024-09-18T15:28:58.196 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:58.196 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:28:58.196 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:58.196 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528379 2024-09-18T15:28:58.702 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubun//home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:58.702 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-455266533384 2024-09-18T15:28:58.702 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:28:58.703 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:58.703 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-455266533384 2024-09-18T15:28:58.703 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=455266533384 2024-09-18T15:28:58.703 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 455266533384' 2024-09-18T15:28:58.703 INFO:tasks.workunit.client.0.smithi046.stderr:307232: waiting osd.9 seq 455266533384 2024-09-18T15:28:58.703 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:28:58.703 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 455266533385 -lt 455266533384 2024-09-18T15:28:58.703 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:28:58.703 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:28:58.703 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:28:58.703 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:28:58.703 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:28:58.704 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:28:58.704 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:28:58.704 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:28:58.704 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:28:58.704 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:28:58.704 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:28:58.704 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:28:58.704 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:28:58.704 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:28:58.704 INFO:tasks.workunit.client.0.smithi046.stderr:307232: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:28:58.705 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:28:58.705 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:28:58.705 INFO:tasks.workunit.client.0.smithi046.stderr:307232: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:28:58.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:28:58.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:28:58.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 307230 2024-09-18T15:28:59.998 INFO:tasks.workunit.client.0.smithi046.stderr:tu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528379 2024-09-18T15:28:59.998 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528379' 2024-09-18T15:28:59.998 INFO:tasks.workunit.client.0.smithi046.stderr:307234: waiting osd.8 seq 193273528379 2024-09-18T15:28:59.999 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:28:59.999 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528380 -lt 193273528379 2024-09-18T15:28:59.999 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:59.999 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-455266533385 2024-09-18T15:28:59.999 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:59.999 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:28:59.999 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:59.999 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-455266533385 2024-09-18T15:28:59.999 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=455266533385 2024-09-18T15:28:59.999 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 455266533385' 2024-09-18T15:28:59.999 INFO:tasks.workunit.client.0.smithi046.stderr:307234: waiting osd.9 seq 455266533385 2024-09-18T15:28:59.999 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:29:00.000 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 455266533385 -lt 455266533385 2024-09-18T15:29:00.000 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:29:00.000 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:29:00.000 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:29:00.000 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:29:00.000 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:29:00.000 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:29:00.000 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:29:00.000 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:29:00.000 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:29:00.001 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:29:00.001 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:29:00.001 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:29:00.001 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:29:00.001 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:29:00.001 INFO:tasks.workunit.client.0.smithi046.stderr:307234: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:29:00.001 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:29:00.001 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:29:00.001 INFO:tasks.workunit.client.0.smithi046.stderr:307234: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:29:00.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:29:00.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: eval 'pids='\'''\''' 2024-09-18T15:29:00.003 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: pids= 2024-09-18T15:29:00.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: wait_background: return 0 2024-09-18T15:29:00.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: return_code=0 2024-09-18T15:29:00.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-18T15:29:00.003 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-18T15:29:00.003 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-09-18T15:29:00.003 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T15:29:00.003 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-18T15:29:00.003 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:29:00.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: local pg=2.0 2024-09-18T15:29:00.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: repair 2.0 2024-09-18T15:29:00.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-09-18T15:29:00.302 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-09-18T15:29:00.302 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:29:00.302 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:29:00.302 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:29:00.302 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:29:00.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T15:28:22.471989+0000 2024-09-18T15:29:00.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-09-18T15:29:00.951 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-18T15:29:00.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-09-18T15:28:22.471989+0000 2024-09-18T15:29:00.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=2.0 2024-09-18T15:29:00.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:28:22.471989+0000 2024-09-18T15:29:00.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:29:00.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:29:00.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:29:00.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:29:00.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:29:00.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:29:00.962 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:29:00.962 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:29:01.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:28:22.471989+0000 '>' 2024-09-18T15:28:22.471989+0000 2024-09-18T15:29:01.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:29:02.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:29:02.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:29:02.318 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:29:02.318 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:29:02.318 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:29:02.318 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:29:02.318 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:29:02.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:28:22.471989+0000 '>' 2024-09-18T15:28:22.471989+0000 2024-09-18T15:29:02.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:29:03.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:29:03.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:29:03.633 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:29:03.633 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:29:03.633 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:29:03.634 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:29:03.634 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:29:03.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:28:22.471989+0000 '>' 2024-09-18T15:28:22.471989+0000 2024-09-18T15:29:03.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:29:04.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:29:04.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:29:04.950 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:29:04.950 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:29:04.950 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:29:04.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:29:04.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:29:05.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:28:22.471989+0000 '>' 2024-09-18T15:28:22.471989+0000 2024-09-18T15:29:05.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:29:06.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:29:06.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:29:06.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:29:06.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:29:06.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:29:06.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:29:06.274 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:29:06.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:28:22.471989+0000 '>' 2024-09-18T15:28:22.471989+0000 2024-09-18T15:29:06.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:29:07.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:29:07.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:29:07.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:29:07.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:29:07.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:29:07.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:29:07.592 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:29:07.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:29:01.022794+0000 '>' 2024-09-18T15:28:22.471989+0000 2024-09-18T15:29:07.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:29:07.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:207: corrupt_and_repair_two: pids= 2024-09-18T15:29:07.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:29:07.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:29:07.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:29:07.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 310876"' 2024-09-18T15:29:07.991 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 310876' 2024-09-18T15:29:07.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-18T15:29:07.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T15:29:07.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T15:29:07.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 310877"' 2024-09-18T15:29:07.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 310877' 2024-09-18T15:29:07.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: wait_background pids 2024-09-18T15:29:07.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: wait_background: pids=' 310876 310877' 2024-09-18T15:29:07.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: wait_background: return_code=0 2024-09-18T15:29:07.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:29:07.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 310876 2024-09-18T15:29:07.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:29:07.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-18T15:29:07.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/310879: /' 2024-09-18T15:29:07.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/310881: /' 2024-09-18T15:29:11.795 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:29:11.795 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:29:11.795 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-18T15:29:11.795 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:29:11.795 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:29:11.796 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:29:11.796 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:29:11.796 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-18T15:29:11.796 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:29:11.796 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-18T15:29:11.796 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:29:11.796 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:29:11.796 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:29:11.796 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:29:11.796 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:29:11.796 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:29:11.796 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-18T15:29:11.796 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:29:11.797 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:29:11.797 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-18T15:29:11.797 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:29:11.797 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:29:11.797 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-18T15:29:11.797 INFO:tasks.workunit.client.0.smithi046.stderr:310879: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-18T15:29:11.797 INFO:tasks.workunit.client.0.smithi046.stderr:310879: _ 2024-09-18T15:29:11.797 INFO:tasks.workunit.client.0.smithi046.stderr:310879: hinfo_key 2024-09-18T15:29:11.797 INFO:tasks.workunit.client.0.smithi046.stderr:310879: snapset 2024-09-18T15:29:11.797 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-18T15:29:11.797 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:29:11.797 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:29:11.797 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T15:29:11.797 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:29:11.798 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-18T15:29:11.798 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:29:11.802 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:29:11.802 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:29:11.802 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:29:11.802 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-18T15:29:11.802 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-18T15:29:11.802 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:29:11.802 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:29:11.802 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:29:11.802 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:29:11.803 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:29:11.803 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:29:11.803 INFO:tasks.workunit.client.0.smithi046.stderr:310879: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:29:11.803 INFO:tasks.workunit.client.0.smithi046.stderr:310879: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:29:11.803 INFO:tasks.workunit.client.0.smithi046.stderr:310879: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:29:11.803 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:29:11.803 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:29:11.803 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:29:11.803 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:29:11.803 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:29:11.803 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:29:11.803 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:29:11.803 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:29:11.803 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:29:11.804 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:29:11.804 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:29:11.804 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-18T15:29:11.804 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T15:29:11.804 INFO:tasks.workunit.client.0.smithi046.stderr:310879: start osd.3 2024-09-18T15:29:11.921 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namesp310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:29:11.921 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:29:11.921 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-09-18T15:29:11.921 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:29:11.921 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-18T15:29:11.922 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:29:11.922 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:29:11.922 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-09-18T15:29:11.922 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:29:11.922 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-18T15:29:11.922 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:29:11.922 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:29:11.922 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:29:11.922 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:29:11.922 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:29:11.922 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:29:11.922 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-18T15:29:11.923 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:29:11.923 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:29:11.923 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-09-18T15:29:11.923 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:29:11.923 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-18T15:29:11.923 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-09-18T15:29:11.923 INFO:tasks.workunit.client.0.smithi046.stderr:310881: _ 2024-09-18T15:29:11.923 INFO:tasks.workunit.client.0.smithi046.stderr:310881: hinfo_key 2024-09-18T15:29:11.923 INFO:tasks.workunit.client.0.smithi046.stderr:310881: snapset 2024-09-18T15:29:11.923 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-18T15:29:11.923 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:29:11.923 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:29:11.923 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-09-18T15:29:11.924 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:29:11.924 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-18T15:29:11.924 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:29:11.924 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:29:11.934 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:29:11.934 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:29:11.934 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-18T15:29:11.934 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-18T15:29:11.935 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:29:11.935 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:29:11.935 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:29:11.935 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:29:11.935 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:29:11.935 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:29:11.935 INFO:tasks.workunit.client.0.smithi046.stderr:310881: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:29:11.935 INFO:tasks.workunit.client.0.smithi046.stderr:310881: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:29:11.935 INFO:tasks.workunit.client.0.smithi046.stderr:310881: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:29:11.935 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:29:11.935 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:29:11.935 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:29:11.936 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:29:11.936 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:29:11.936 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:29:11.936 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:29:11.936 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:29:11.936 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:29:11.936 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:29:11.936 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:29:11.936 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-18T15:29:11.936 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-09-18T15:29:11.936 INFO:tasks.workunit.client.0.smithi046.stderr:310881: start osd.5 2024-09-18T15:29:16.120 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=highace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:29:16.120 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-18T15:29:16.120 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T15:29:16.120 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:29:16.120 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:29:16.120 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:29:16.120 INFO:tasks.workunit.client.0.smithi046.stderr:310879: 2024-09-18T15:29:11.824+0000 7efee71a5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:29:16.121 INFO:tasks.workunit.client.0.smithi046.stderr:310879: 2024-09-18T15:29:11.828+0000 7efee71a5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:29:16.121 INFO:tasks.workunit.client.0.smithi046.stderr:310879: 2024-09-18T15:29:11.828+0000 7efee71a5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:29:16.121 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T15:29:16.121 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:29:16.121 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:29:16.121 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:29:16.121 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:29:16.121 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:16.121 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:29:16.121 INFO:tasks.workunit.client.0.smithi046.stderr:310879: 0 2024-09-18T15:29:16.121 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:16.121 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:29:16.121 INFO:tasks.workunit.client.0.smithi046.stderr:310879: 2024-09-18T15:29:12.380+0000 7efee71a5540 -1 Falling back to public interface 2024-09-18T15:29:16.122 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:16.122 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:16.122 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:16.122 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:29:16.122 INFO:tasks.workunit.client.0.smithi046.stderr:310879: 1 2024-09-18T15:29:16.122 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:16.122 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:29:16.122 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:16.122 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:16.122 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:16.122 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:29:16.122 INFO:tasks.workunit.client.0.smithi046.stderr:310879: 2 2024-09-18T15:29:16.122 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:16.122 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:29:16.123 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:16.123 INFO:tasks.workunit.client.0.smithi046.stderr:310879: 2024-09-18T15:29:15.172+0000 7efee71a5540 -1 osd.3 117 log_to_monitors true 2024-09-18T15:29:16.123 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:16.123 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:16.123 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:29:16.123 INFO:tasks.workunit.client.0.smithi046.stderr:310879: 3 2024-09-18T15:29:16.123 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:16.899 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-h_recovery_ops 2024-09-18T15:29:16.899 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-18T15:29:16.899 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-09-18T15:29:16.899 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:29:16.900 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:29:16.900 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:29:16.900 INFO:tasks.workunit.client.0.smithi046.stderr:310881: 2024-09-18T15:29:11.952+0000 7f78be924540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:29:16.900 INFO:tasks.workunit.client.0.smithi046.stderr:310881: 2024-09-18T15:29:11.956+0000 7f78be924540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:29:16.900 INFO:tasks.workunit.client.0.smithi046.stderr:310881: 2024-09-18T15:29:11.956+0000 7f78be924540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:29:16.900 INFO:tasks.workunit.client.0.smithi046.stderr:310881: 2024-09-18T15:29:12.240+0000 7f78be924540 -1 Falling back to public interface 2024-09-18T15:29:16.900 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-09-18T15:29:16.900 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:29:16.900 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T15:29:16.901 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:29:16.901 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:29:16.901 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:16.901 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:29:16.901 INFO:tasks.workunit.client.0.smithi046.stderr:310881: 0 2024-09-18T15:29:16.901 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:29:16.901 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:16.901 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:16.901 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:16.901 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:16.901 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:29:16.901 INFO:tasks.workunit.client.0.smithi046.stderr:310881: 1 2024-09-18T15:29:16.902 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:16.902 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:29:16.902 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:16.902 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:16.902 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:16.902 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:29:16.902 INFO:tasks.workunit.client.0.smithi046.stderr:310881: 2 2024-09-18T15:29:16.902 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:16.902 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:29:16.903 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:16.903 INFO:tasks.workunit.client.0.smithi046.stderr:310881: 2024-09-18T15:29:15.592+0000 7f78be924540 -1 osd.5 117 log_to_monitors true 2024-09-18T15:29:16.903 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:16.903 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:16.903 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:29:16.903 INFO:tasks.workunit.client.0.smithi046.stderr:310881: 3 2024-09-18T15:29:16.903 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:16.903 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:29:18.122 INFO:tasks.workunit.client.0.smithi046.stderr:310881: osd.5 up in weight 1 up_from 122 up_thru 116elpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:29:18.122 INFO:tasks.workunit.client.0.smithi046.stderr:310879: osd.3 up in weight 1 up_from 121 up_thru 116 down_at 118 last_clean_interval [113,117) [v2:127.0.0.1:6842/311686,v1:127.0.0.1:6843/311686] [v2:127.0.0.1:6844/311686,v1:127.0.0.1:6845/311686] exists,up 105fcb65-1d2b-4358-9b52-7d1b65a95f69 2024-09-18T15:29:18.122 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:29:18.122 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:29:18.122 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:29:18.122 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:29:18.122 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:29:18.122 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:29:18.122 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:29:18.122 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:29:18.123 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:29:18.123 INFO:tasks.workunit.client.0.smithi046.stderr:310879: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:29:18.123 INFO:tasks.workunit.client.0.smithi046.stderr:310879: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:29:18.123 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:29:18.123 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:29:18.123 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:29:18.123 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:29:18.123 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:29:18.123 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:29:18.123 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:29:18.123 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:29:18.123 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:29:18.123 INFO:tasks.workunit.client.0.smithi046.stderr:310879: 1 2024-09-18T15:29:18.124 INFO:tasks.workunit.client.0.smithi046.stderr:310879: 2 2024-09-18T15:29:18.124 INFO:tasks.workunit.client.0.smithi046.stderr:310879: 3 2024-09-18T15:29:18.124 INFO:tasks.workunit.client.0.smithi046.stderr:310879: 4 2024-09-18T15:29:18.124 INFO:tasks.workunit.client.0.smithi046.stderr:310879: 5 2024-09-18T15:29:18.124 INFO:tasks.workunit.client.0.smithi046.stderr:310879: 6 2024-09-18T15:29:18.124 INFO:tasks.workunit.client.0.smithi046.stderr:310879: 7 2024-09-18T15:29:18.124 INFO:tasks.workunit.client.0.smithi046.stderr:310879: 8 2024-09-18T15:29:18.124 INFO:tasks.workunit.client.0.smithi046.stderr:310879: 9' 2024-09-18T15:29:18.124 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:29:18.124 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:29:18.124 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:29:18.124 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836603 2024-09-18T15:29:18.124 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836603 2024-09-18T15:29:18.125 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836603' 2024-09-18T15:29:18.125 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:29:18.125 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:29:18.125 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673073 2024-09-18T15:29:18.125 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673073 2024-09-18T15:29:18.125 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836603 1-42949673073' 2024-09-18T15:29:18.346 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: f down_at 118 last_clean_interval [116,117) [v2:127.0.0.1:6826/311731,v1:127.0.0.1:6827/311731] [v2:127.0.0.1:6828/311731,v1:127.0.0.1:6829/311731] exists,up 63244125-a488-402d-9bdf-d4f6f2b1fbf0 2024-09-18T15:29:18.346 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:29:18.346 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:29:18.346 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:29:18.346 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:29:18.346 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:29:18.346 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:29:18.346 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:29:18.346 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:29:18.346 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:29:18.346 INFO:tasks.workunit.client.0.smithi046.stderr:310881: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:29:18.347 INFO:tasks.workunit.client.0.smithi046.stderr:310881: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:29:18.347 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:29:18.347 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:29:18.347 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:29:18.347 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:29:18.347 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:29:18.347 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:29:18.347 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:29:18.347 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:29:18.347 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:29:18.347 INFO:tasks.workunit.client.0.smithi046.stderr:310881: 1 2024-09-18T15:29:18.347 INFO:tasks.workunit.client.0.smithi046.stderr:310881: 2 2024-09-18T15:29:18.347 INFO:tasks.workunit.client.0.smithi046.stderr:310881: 3 2024-09-18T15:29:18.348 INFO:tasks.workunit.client.0.smithi046.stderr:310881: 4 2024-09-18T15:29:18.348 INFO:tasks.workunit.client.0.smithi046.stderr:310881: 5 2024-09-18T15:29:18.348 INFO:tasks.workunit.client.0.smithi046.stderr:310881: 6 2024-09-18T15:29:18.348 INFO:tasks.workunit.client.0.smithi046.stderr:310881: 7 2024-09-18T15:29:18.348 INFO:tasks.workunit.client.0.smithi046.stderr:310881: 8 2024-09-18T15:29:18.348 INFO:tasks.workunit.client.0.smithi046.stderr:310881: 9' 2024-09-18T15:29:18.348 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:29:18.348 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:29:18.348 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:29:18.349 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836604 2024-09-18T15:29:18.349 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836604 2024-09-18T15:29:18.349 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836604' 2024-09-18T15:29:18.349 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:29:18.349 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:29:18.349 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673075 2024-09-18T15:29:18.349 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673075 2024-09-18T15:29:18.349 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836604 1-42949673075' 2024-09-18T15:29:18.349 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:29:19.062 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helplush_pg_stats: for osd in $ids 2024-09-18T15:29:19.062 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:29:19.062 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509550 2024-09-18T15:29:19.062 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509550 2024-09-18T15:29:19.062 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836603 1-42949673073 2-64424509550' 2024-09-18T15:29:19.062 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:29:19.062 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:29:19.062 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=519691042818 2024-09-18T15:29:19.062 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 519691042818 2024-09-18T15:29:19.063 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836603 1-42949673073 2-64424509550 3-519691042818' 2024-09-18T15:29:19.063 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:29:19.063 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:29:19.063 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182495 2024-09-18T15:29:19.063 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182495 2024-09-18T15:29:19.063 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836603 1-42949673073 2-64424509550 3-519691042818 4-107374182495' 2024-09-18T15:29:19.063 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:29:19.063 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:29:19.063 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=523986010114 2024-09-18T15:29:19.063 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 523986010114 2024-09-18T15:29:19.063 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836603 1-42949673073 2-64424509550 3-519691042818 4-107374182495 5-523986010114' 2024-09-18T15:29:19.063 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:29:19.064 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:29:19.064 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855442 2024-09-18T15:29:19.064 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855442 2024-09-18T15:29:19.064 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836603 1-42949673073 2-64424509550 3-519691042818 4-107374182495 5-523986010114 6-150323855442' 2024-09-18T15:29:19.064 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:29:19.064 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:29:19.064 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691913 2024-09-18T15:29:19.064 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691913 2024-09-18T15:29:19.064 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836603 1-42949673073 2-64424509550 3-519691042818 4-107374182495 5-523986010114 6-150323855442 7-171798691913' 2024-09-18T15:29:19.065 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:29:19.330 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: fluers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:29:19.331 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509551 2024-09-18T15:29:19.331 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509551 2024-09-18T15:29:19.331 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836604 1-42949673075 2-64424509551' 2024-09-18T15:29:19.331 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:29:19.331 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:29:19.331 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=519691042819 2024-09-18T15:29:19.331 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 519691042819 2024-09-18T15:29:19.331 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836604 1-42949673075 2-64424509551 3-519691042819' 2024-09-18T15:29:19.331 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:29:19.331 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:29:19.331 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182496 2024-09-18T15:29:19.332 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182496 2024-09-18T15:29:19.332 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836604 1-42949673075 2-64424509551 3-519691042819 4-107374182496' 2024-09-18T15:29:19.332 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:29:19.332 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:29:19.332 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=523986010115 2024-09-18T15:29:19.332 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 523986010115 2024-09-18T15:29:19.332 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836604 1-42949673075 2-64424509551 3-519691042819 4-107374182496 5-523986010115' 2024-09-18T15:29:19.332 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:29:19.332 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T15:29:19.332 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855443 2024-09-18T15:29:19.332 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855443 2024-09-18T15:29:19.332 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836604 1-42949673075 2-64424509551 3-519691042819 4-107374182496 5-523986010115 6-150323855443' 2024-09-18T15:29:19.332 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:29:19.333 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T15:29:19.333 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691914 2024-09-18T15:29:19.333 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691914 2024-09-18T15:29:19.333 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836604 1-42949673075 2-64424509551 3-519691042819 4-107374182496 5-523986010115 6-150323855443 7-171798691914' 2024-09-18T15:29:19.333 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:29:19.333 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:29:19.985 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalosh_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T15:29:19.985 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528385 2024-09-18T15:29:19.985 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528385 2024-09-18T15:29:19.985 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836603 1-42949673073 2-64424509550 3-519691042818 4-107374182495 5-523986010114 6-150323855442 7-171798691913 8-193273528385' 2024-09-18T15:29:19.986 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:29:19.986 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:29:19.986 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=455266533391 2024-09-18T15:29:19.986 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 455266533391 2024-09-18T15:29:19.986 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836603 1-42949673073 2-64424509550 3-519691042818 4-107374182495 5-523986010114 6-150323855442 7-171798691913 8-193273528385 9-455266533391' 2024-09-18T15:29:19.986 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:29:19.986 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836603 2024-09-18T15:29:19.986 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:29:19.986 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:29:19.986 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836603 2024-09-18T15:29:19.986 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:29:19.986 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836603 2024-09-18T15:29:19.986 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836603' 2024-09-18T15:29:19.987 INFO:tasks.workunit.client.0.smithi046.stderr:310879: waiting osd.0 seq 21474836603 2024-09-18T15:29:19.987 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:29:19.987 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836605 -lt 21474836603 2024-09-18T15:29:19.987 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:29:19.987 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673073 2024-09-18T15:29:19.987 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:29:19.987 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:29:19.987 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673073 2024-09-18T15:29:19.987 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:29:19.987 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673073 2024-09-18T15:29:19.987 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673073' 2024-09-18T15:29:19.987 INFO:tasks.workunit.client.0.smithi046.stderr:310879: waiting osd.1 seq 42949673073 2024-09-18T15:29:19.987 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:29:19.987 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673075 -lt 42949673073 2024-09-18T15:29:19.988 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:29:19.988 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509550 2024-09-18T15:29:19.988 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:29:20.101 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/cepne/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528386 2024-09-18T15:29:20.101 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528386 2024-09-18T15:29:20.101 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836604 1-42949673075 2-64424509551 3-519691042819 4-107374182496 5-523986010115 6-150323855443 7-171798691914 8-193273528386' 2024-09-18T15:29:20.101 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:29:20.101 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T15:29:20.101 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=455266533392 2024-09-18T15:29:20.101 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 455266533392 2024-09-18T15:29:20.101 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836604 1-42949673075 2-64424509551 3-519691042819 4-107374182496 5-523986010115 6-150323855443 7-171798691914 8-193273528386 9-455266533392' 2024-09-18T15:29:20.101 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:29:20.101 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836604 2024-09-18T15:29:20.102 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:29:20.102 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:29:20.102 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836604 2024-09-18T15:29:20.102 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:29:20.102 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836604 2024-09-18T15:29:20.102 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836604' 2024-09-18T15:29:20.102 INFO:tasks.workunit.client.0.smithi046.stderr:310881: waiting osd.0 seq 21474836604 2024-09-18T15:29:20.102 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:29:20.102 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836605 -lt 21474836604 2024-09-18T15:29:20.102 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:29:20.102 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:29:20.102 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673075 2024-09-18T15:29:20.102 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:29:20.102 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673075 2024-09-18T15:29:20.102 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:29:20.103 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673075 2024-09-18T15:29:20.103 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673075' 2024-09-18T15:29:20.103 INFO:tasks.workunit.client.0.smithi046.stderr:310881: waiting osd.1 seq 42949673075 2024-09-18T15:29:20.103 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:29:20.103 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673075 -lt 42949673075 2024-09-18T15:29:20.103 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:29:20.103 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:29:20.103 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509551 2024-09-18T15:29:20.103 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:29:21.003 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standaloneh-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:29:21.003 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509550 2024-09-18T15:29:21.003 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:29:21.003 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509550 2024-09-18T15:29:21.003 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509550' 2024-09-18T15:29:21.004 INFO:tasks.workunit.client.0.smithi046.stderr:310879: waiting osd.2 seq 64424509550 2024-09-18T15:29:21.004 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:29:21.004 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509551 -lt 64424509550 2024-09-18T15:29:21.004 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:29:21.004 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:29:21.004 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-519691042818 2024-09-18T15:29:21.004 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:29:21.004 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-519691042818 2024-09-18T15:29:21.004 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:29:21.004 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=519691042818 2024-09-18T15:29:21.004 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 519691042818' 2024-09-18T15:29:21.004 INFO:tasks.workunit.client.0.smithi046.stderr:310879: waiting osd.3 seq 519691042818 2024-09-18T15:29:21.005 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:29:21.005 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 519691042819 -lt 519691042818 2024-09-18T15:29:21.005 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:29:21.005 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182495 2024-09-18T15:29:21.005 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:29:21.005 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:29:21.005 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:29:21.005 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182495 2024-09-18T15:29:21.005 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182495 2024-09-18T15:29:21.005 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182495' 2024-09-18T15:29:21.005 INFO:tasks.workunit.client.0.smithi046.stderr:310879: waiting osd.4 seq 107374182495 2024-09-18T15:29:21.005 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:29:21.005 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182496 -lt 107374182495 2024-09-18T15:29:21.006 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:29:21.006 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-523986010114 2024-09-18T15:29:21.006 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:29:21.006 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:29:21.006 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-523986010114 2024-09-18T15:29:21.006 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:29:21.006 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=523986010114 2024-09-18T15:29:21.030 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:29:21.030 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509551 2024-09-18T15:29:21.031 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509551 2024-09-18T15:29:21.031 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509551' 2024-09-18T15:29:21.031 INFO:tasks.workunit.client.0.smithi046.stderr:310881: waiting osd.2 seq 64424509551 2024-09-18T15:29:21.031 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:29:21.031 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509551 -lt 64424509551 2024-09-18T15:29:21.031 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:29:21.031 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-519691042819 2024-09-18T15:29:21.031 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:29:21.031 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:29:21.031 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-519691042819 2024-09-18T15:29:21.031 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:29:21.031 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=519691042819 2024-09-18T15:29:21.032 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 519691042819' 2024-09-18T15:29:21.032 INFO:tasks.workunit.client.0.smithi046.stderr:310881: waiting osd.3 seq 519691042819 2024-09-18T15:29:21.032 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:29:21.032 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 519691042819 -lt 519691042819 2024-09-18T15:29:21.032 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:29:21.032 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182496 2024-09-18T15:29:21.032 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:29:21.032 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:29:21.032 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182496 2024-09-18T15:29:21.032 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:29:21.033 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182496 2024-09-18T15:29:21.033 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182496' 2024-09-18T15:29:21.033 INFO:tasks.workunit.client.0.smithi046.stderr:310881: waiting osd.4 seq 107374182496 2024-09-18T15:29:21.033 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:29:21.033 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182496 -lt 107374182496 2024-09-18T15:29:21.033 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:29:21.034 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-523986010115 2024-09-18T15:29:21.034 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:29:21.034 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:29:21.034 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-523986010115 2024-09-18T15:29:21.034 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:29:21.034 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=523986010115 2024-09-18T15:29:22.423 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 523986010115' 2024-09-18T15:29:22.423 INFO:tasks.workunit.client.0.smithi046.stderr:310881: waiting osd.5 seq 523986010115 2024-09-18T15:29:22.423 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:29:22.423 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 523986010115 -lt 523986010115 2024-09-18T15:29:22.423 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:29:22.423 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855443 2024-09-18T15:29:22.424 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:29:22.424 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:29:22.424 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855443 2024-09-18T15:29:22.424 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:29:22.424 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855443 2024-09-18T15:29:22.424 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855443' 2024-09-18T15:29:22.424 INFO:tasks.workunit.client.0.smithi046.stderr:310881: waiting osd.6 seq 150323855443 2024-09-18T15:29:22.424 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:29:22.424 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855443 -lt 150323855443 2024-09-18T15:29:22.424 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:29:22.424 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691914 2024-09-18T15:29:22.424 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:29:22.424 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:29:22.425 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:29:22.425 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691914 2024-09-18T15:29:22.425 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691914 2024-09-18T15:29:22.425 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691914' 2024-09-18T15:29:22.425 INFO:tasks.workunit.client.0.smithi046.stderr:310881: waiting osd.7 seq 171798691914 2024-09-18T15:29:22.425 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:29:22.425 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691915 -lt 171798691914 2024-09-18T15:29:22.425 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:29:22.425 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528386 2024-09-18T15:29:22.425 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:29:22.425 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:29:22.425 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528386 2024-09-18T15:29:22.426 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:29:22.426 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528386 2024-09-18T15:29:22.426 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528386' 2024-09-18T15:29:22.426 INFO:tasks.workunit.client.0.smithi046.stderr:310881: waiting osd.8 seq 193273528386 2024-09-18T15:29:22.426 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:29:22.426 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528387 -lt 193273528386 2024-09-18T15:29:22.426 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $se/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 523986010114' 2024-09-18T15:29:22.426 INFO:tasks.workunit.client.0.smithi046.stderr:310879: waiting osd.5 seq 523986010114 2024-09-18T15:29:22.426 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:29:22.426 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 523986010115 -lt 523986010114 2024-09-18T15:29:22.426 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:29:22.427 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:29:22.427 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855442 2024-09-18T15:29:22.427 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T15:29:22.427 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855442 2024-09-18T15:29:22.427 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:29:22.427 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855442 2024-09-18T15:29:22.427 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855442' 2024-09-18T15:29:22.427 INFO:tasks.workunit.client.0.smithi046.stderr:310879: waiting osd.6 seq 150323855442 2024-09-18T15:29:22.427 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T15:29:22.427 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855443 -lt 150323855442 2024-09-18T15:29:22.427 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:29:22.427 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691913 2024-09-18T15:29:22.427 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:29:22.428 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T15:29:22.428 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691913 2024-09-18T15:29:22.428 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:29:22.428 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691913 2024-09-18T15:29:22.428 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691913' 2024-09-18T15:29:22.428 INFO:tasks.workunit.client.0.smithi046.stderr:310879: waiting osd.7 seq 171798691913 2024-09-18T15:29:22.428 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T15:29:22.428 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691915 -lt 171798691913 2024-09-18T15:29:22.428 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:29:22.428 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528385 2024-09-18T15:29:22.428 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:29:22.428 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T15:29:22.428 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528385 2024-09-18T15:29:22.428 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:29:22.429 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528385 2024-09-18T15:29:22.429 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528385' 2024-09-18T15:29:22.429 INFO:tasks.workunit.client.0.smithi046.stderr:310879: waiting osd.8 seq 193273528385 2024-09-18T15:29:22.429 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T15:29:22.429 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528387 -lt 193273528385 2024-09-18T15:29:23.874 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /qs 2024-09-18T15:29:23.874 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-455266533392 2024-09-18T15:29:23.874 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:29:23.875 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:29:23.875 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-455266533392 2024-09-18T15:29:23.875 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:29:23.875 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=455266533392 2024-09-18T15:29:23.875 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 455266533392' 2024-09-18T15:29:23.875 INFO:tasks.workunit.client.0.smithi046.stderr:310881: waiting osd.9 seq 455266533392 2024-09-18T15:29:23.875 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:29:23.875 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 455266533392 -lt 455266533392 2024-09-18T15:29:23.875 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:29:23.875 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:29:23.875 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:29:23.875 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:29:23.875 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:29:23.876 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:29:23.876 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:29:23.876 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:29:23.876 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:29:23.876 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:29:23.876 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:29:23.876 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:29:23.876 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:29:23.876 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:29:23.876 INFO:tasks.workunit.client.0.smithi046.stderr:310881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:29:23.876 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:29:23.876 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:29:23.876 INFO:tasks.workunit.client.0.smithi046.stderr:310881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:29:23.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:29:23.878 INFO:tasks.workunit.client.0.smithi046.stderr:home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:29:23.878 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:29:23.879 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-455266533391 2024-09-18T15:29:23.879 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T15:29:23.879 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-455266533391 2024-09-18T15:29:23.879 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:29:23.879 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=455266533391 2024-09-18T15:29:23.879 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 455266533391' 2024-09-18T15:29:23.879 INFO:tasks.workunit.client.0.smithi046.stderr:310879: waiting osd.9 seq 455266533391 2024-09-18T15:29:23.879 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T15:29:23.879 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 455266533392 -lt 455266533391 2024-09-18T15:29:23.879 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:29:23.879 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:29:23.879 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:29:23.880 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:29:23.880 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:29:23.880 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:29:23.880 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:29:23.880 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:29:23.880 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:29:23.880 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:29:23.880 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:29:23.880 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:29:23.880 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:29:23.880 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:29:23.880 INFO:tasks.workunit.client.0.smithi046.stderr:310879: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:29:23.880 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:29:23.880 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:29:23.881 INFO:tasks.workunit.client.0.smithi046.stderr:310879: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:29:23.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T15:29:23.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T15:29:23.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 310877 2024-09-18T15:29:23.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: eval 'pids='\'''\''' 2024-09-18T15:29:23.882 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: pids= 2024-09-18T15:29:23.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: wait_background: return 0 2024-09-18T15:29:23.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: return_code=0 2024-09-18T15:29:23.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-18T15:29:23.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:214: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-18T15:29:23.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-18T15:29:23.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T15:29:23.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:29:23.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:29:23.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:29:23.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:29:23.973 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:29:23.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:29:23.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:29:23.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:29:24.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:29:24.302 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:29:24.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:29:24.305 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:29:24.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:29:24.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:29:24.308 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:29:24.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:29:24.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:29:24.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:29:24.311 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:29:24.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:29:24.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:29:24.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:29:24.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:29:24.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:29:24.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:29:24.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:29:24.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:29:24.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:29:24.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T15:29:24.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T15:29:24.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T15:29:24.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T15:29:24.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:29:24.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:29:24.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:29:24.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:29:24.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:29:24.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:29:24.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:29:24.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:29:24.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:29:24.393 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:29:24.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:29:24.396 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:29:24.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:29:24.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:29:24.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:29:24.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:29:24.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:29:24.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:29:24.401 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:29:24.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:29:24.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:29:24.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:29:24.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:29:24.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:29:24.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:29:24.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:29:24.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:29:24.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:29:24.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T15:29:24.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:29:24.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:29:24.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:29:24.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T15:29:24.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:29:24.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:29:24.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:29:24.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T15:29:24.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_corrupt_and_repair_replicated td/osd-scrub-repair 2024-09-18T15:29:24.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:93: TEST_corrupt_and_repair_replicated: local dir=td/osd-scrub-repair 2024-09-18T15:29:24.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:94: TEST_corrupt_and_repair_replicated: local poolname=rbd 2024-09-18T15:29:24.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:96: TEST_corrupt_and_repair_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-09-18T15:29:24.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T15:29:24.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:29:24.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:29:24.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:29:24.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T15:29:24.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-09-18T15:29:24.799 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:29:24.799 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:29:24.799 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:29:24.800 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:29:24.800 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:29:24.800 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:29:24.800 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:29:24.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-18T15:29:25.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:29:25.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:29:25.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:29:25.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:29:25.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:29:25.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:29:25.074 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:29:25.075 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:29:25.075 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:29:25.075 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:29:25.076 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:29:25.076 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:29:25.076 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:29:25.077 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:29:25.077 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T15:29:25.174 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:29:25.174 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:29:25.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:29:25.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:29:25.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:29:25.176 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:29:25.176 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:29:25.176 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:29:25.177 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:29:25.177 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:29:25.177 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:29:25.177 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:29:25.177 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:29:25.178 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T15:29:25.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:97: TEST_corrupt_and_repair_replicated: run_mgr td/osd-scrub-repair x 2024-09-18T15:29:25.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T15:29:25.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:29:25.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:29:25.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:29:25.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T15:29:25.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:29:26.189 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:29:26.189 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:29:26.190 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:29:26.190 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:29:26.190 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:29:26.190 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:29:26.190 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:29:26.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:29:26.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:29:26.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:98: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 0 2024-09-18T15:29:26.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:29:26.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:29:26.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:29:26.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:29:26.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:29:26.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:29:26.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:29:26.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:29:26.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:29:26.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:29:26.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:29:26.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:29:26.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:29:26.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:29:26.232 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:29:26.232 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:29:26.232 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:29:26.232 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:29:26.232 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:29:26.232 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:29:26.234 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:29:26.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:29:26.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:29:26.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:29:26.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:29:26.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:29:26.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:29:26.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:29:26.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:29:26.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:29:26.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:29:26.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:29:26.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:29:26.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:29:26.236 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:29:26.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d8472018-4074-42c2-b75f-ab3e86a0690e 2024-09-18T15:29:26.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d8472018-4074-42c2-b75f-ab3e86a0690e' 2024-09-18T15:29:26.238 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 d8472018-4074-42c2-b75f-ab3e86a0690e 2024-09-18T15:29:26.238 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:29:26.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDW8epmF7gJDxAAy9sc4PeOcOSbYgv3Xiatxw== 2024-09-18T15:29:26.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDW8epmF7gJDxAAy9sc4PeOcOSbYgv3Xiatxw=="}' 2024-09-18T15:29:26.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d8472018-4074-42c2-b75f-ab3e86a0690e -i td/osd-scrub-repair/0/new.json 2024-09-18T15:29:26.531 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:29:26.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-18T15:29:26.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDW8epmF7gJDxAAy9sc4PeOcOSbYgv3Xiatxw== --osd-uuid d8472018-4074-42c2-b75f-ab3e86a0690e 2024-09-18T15:29:26.580 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:29:26.568+0000 7f37bf44b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:29:26.624 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:29:26.612+0000 7f37bf44b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:29:26.626 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:29:26.616+0000 7f37bf44b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:29:26.627 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:29:26.616+0000 7f37bf44b540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:29:26.627 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:29:26.616+0000 7f37bf44b540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:29:26.627 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:29:26.616+0000 7f37bf44b540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:29:26.672 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:29:26.660+0000 7f37bf44b540 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-18T15:29:32.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T15:29:32.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:29:32.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:29:32.920 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T15:29:32.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:29:33.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:29:33.312 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T15:29:33.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:29:33.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:29:33.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:29:33.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:29:33.343 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:29:33.332+0000 7fc096194540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:29:33.346 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:29:33.336+0000 7fc096194540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:29:33.347 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:29:33.336+0000 7fc096194540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:29:33.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:29:33.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:29:33.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:29:33.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:29:33.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:29:33.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:33.662 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:29:33.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:29:33.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:33.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:29:33.899 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:29:33.888+0000 7fc096194540 -1 Falling back to public interface 2024-09-18T15:29:33.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:34.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:34.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:34.969 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:29:34.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:29:34.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:34.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:29:35.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:36.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:36.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:36.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:29:36.281 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:29:36.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:36.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:29:36.418 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:29:36.408+0000 7fc096194540 -1 osd.0 0 log_to_monitors true 2024-09-18T15:29:36.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:37.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:37.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:37.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:29:37.601 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:29:37.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:37.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:29:37.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:38.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:38.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:38.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:29:38.923 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:29:38.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:38.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:29:39.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:40.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:40.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:40.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:29:40.394 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:29:40.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:40.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:29:40.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:41.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:41.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:41.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:29:41.699 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:29:41.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:41.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:29:42.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:43.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:43.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:43.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:29:43.014 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:29:43.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:43.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:29:43.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:44.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:44.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:44.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:29:44.326 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:29:44.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:44.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:29:44.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:45.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:45.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:45.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:29:45.646 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:29:45.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:45.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:29:45.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:46.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:46.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:46.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:29:46.985 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:29:46.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:46.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:29:47.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:48.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:48.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:48.328 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:29:48.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:29:48.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:48.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:29:48.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:49.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:49.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:49.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:29:49.642 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:29:49.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:49.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:29:50.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:51.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:51.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:51.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:29:51.116 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:29:51.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:51.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:29:51.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:52.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:52.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:52.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:29:52.680 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:29:52.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:52.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:29:53.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:54.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:54.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:54.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:29:54.062 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:29:54.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:54.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:29:54.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:55.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:55.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:55.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:29:55.751 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:29:55.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:55.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:29:56.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:57.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:57.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:57.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:29:57.356 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:29:57.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:57.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:29:58.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:59.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:59.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:59.363 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:29:59.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:29:59.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:59.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:30:00.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:01.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:01.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:01.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:30:01.202 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:30:01.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:01.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:30:01.543 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/315473,v1:127.0.0.1:6803/315473] [v2:127.0.0.1:6804/315473,v1:127.0.0.1:6805/315473] exists,up d8472018-4074-42c2-b75f-ab3e86a0690e 2024-09-18T15:30:01.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:30:01.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:30:01.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:30:01.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:99: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 1 2024-09-18T15:30:01.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:30:01.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:30:01.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:30:01.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:30:01.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:30:01.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:30:01.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:30:01.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:30:01.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:30:01.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:30:01.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:30:01.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:30:01.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:30:01.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:30:01.546 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:30:01.547 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:30:01.547 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:30:01.547 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:30:01.547 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:30:01.547 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:30:01.547 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:30:01.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:30:01.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:30:01.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:30:01.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:30:01.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:30:01.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:30:01.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:30:01.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:30:01.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:30:01.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:30:01.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:30:01.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:30:01.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:30:01.548 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:30:01.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5775dd03-c507-469b-9a6a-e2fe091ddc66 2024-09-18T15:30:01.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 5775dd03-c507-469b-9a6a-e2fe091ddc66' 2024-09-18T15:30:01.549 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 5775dd03-c507-469b-9a6a-e2fe091ddc66 2024-09-18T15:30:01.550 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:30:01.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD58epmxnrdIRAAJ685mJdsD+48qmhEBav7iQ== 2024-09-18T15:30:01.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD58epmxnrdIRAAJ685mJdsD+48qmhEBav7iQ=="}' 2024-09-18T15:30:01.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5775dd03-c507-469b-9a6a-e2fe091ddc66 -i td/osd-scrub-repair/1/new.json 2024-09-18T15:30:01.987 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:30:02.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-18T15:30:02.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD58epmxnrdIRAAJ685mJdsD+48qmhEBav7iQ== --osd-uuid 5775dd03-c507-469b-9a6a-e2fe091ddc66 2024-09-18T15:30:02.041 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:30:02.036+0000 7fad88397540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:30:02.045 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:30:02.040+0000 7fad88397540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:30:02.047 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:30:02.040+0000 7fad88397540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:30:02.048 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:30:02.040+0000 7fad88397540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:30:02.048 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:30:02.040+0000 7fad88397540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:30:02.048 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:30:02.040+0000 7fad88397540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:30:02.114 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:30:02.108+0000 7fad88397540 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-18T15:30:08.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-18T15:30:08.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:30:08.756 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T15:30:08.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:30:08.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:30:09.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:30:09.139 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T15:30:09.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:30:09.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:30:09.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:30:09.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:30:09.170 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:30:09.164+0000 7fbb8c9ea540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:30:09.173 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:30:09.168+0000 7fbb8c9ea540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:30:09.174 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:30:09.168+0000 7fbb8c9ea540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:30:09.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:30:09.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:30:09.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:30:09.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:30:09.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:30:09.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:09.446 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:30:09.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:30:09.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:09.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:30:09.715 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:30:09.708+0000 7fbb8c9ea540 -1 Falling back to public interface 2024-09-18T15:30:09.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:10.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:10.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:10.809 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:30:10.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:30:10.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:10.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:30:11.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:12.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:12.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:12.130 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:30:12.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:30:12.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:12.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:30:12.345 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:30:12.328+0000 7fbb8c9ea540 -1 osd.1 0 log_to_monitors true 2024-09-18T15:30:12.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:13.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:13.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:13.525 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:30:13.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:30:13.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:13.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:30:13.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:14.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:14.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:14.965 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:30:14.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:30:14.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:14.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:30:15.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:16.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:16.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:16.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:30:16.285 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:30:16.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:16.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:30:16.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:17.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:17.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:17.713 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:30:17.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:30:17.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:17.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:30:18.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:19.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:19.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:19.084 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:30:19.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:30:19.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:19.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:30:19.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:20.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:20.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:20.478 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:30:20.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:30:20.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:20.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:30:20.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:21.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:21.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:21.809 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:30:21.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:30:21.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:21.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:30:22.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:23.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:23.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:23.134 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:30:23.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:30:23.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:23.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:30:23.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:24.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:24.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:24.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:30:24.494 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:30:24.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:24.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:30:24.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:25.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:25.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:25.879 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:30:25.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:30:25.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:25.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:30:26.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:27.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:27.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:27.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:30:27.261 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:30:27.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:27.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:30:27.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:28.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:28.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:28.740 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:30:28.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:30:28.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:28.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:30:29.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:30.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:30.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:30.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:30:30.288 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:30:30.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:30.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:30:30.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:31.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:31.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:31.809 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:30:31.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:30:31.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:31.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:30:32.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:33.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:33.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:33.367 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:30:33.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:30:33.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:33.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:30:33.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:34.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:34.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:34.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:30:34.719 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:30:34.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:34.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:30:35.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:36.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:36.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:36.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:30:36.271 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:30:36.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:36.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:30:36.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:37.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:37.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:37.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-18T15:30:37.677 INFO:tasks.workunit.client.0.smithi046.stdout:20 2024-09-18T15:30:37.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:37.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:30:38.024 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/317278,v1:127.0.0.1:6811/317278] [v2:127.0.0.1:6812/317278,v1:127.0.0.1:6813/317278] exists,up 5775dd03-c507-469b-9a6a-e2fe091ddc66 2024-09-18T15:30:38.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:30:38.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:30:38.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:30:38.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:100: TEST_corrupt_and_repair_replicated: create_rbd_pool 2024-09-18T15:30:38.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T15:30:38.390 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' does not exist 2024-09-18T15:30:38.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T15:30:38.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T15:30:38.863 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' created 2024-09-18T15:30:38.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:30:39.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T15:30:42.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:101: TEST_corrupt_and_repair_replicated: wait_for_clean 2024-09-18T15:30:42.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:30:42.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:30:42.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:30:42.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:30:42.170 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:30:42.171 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:30:42.171 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:30:42.172 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:30:42.172 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:30:42.172 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:30:42.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:30:42.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:30:42.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:30:42.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:30:42.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:30:42.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:30:42.720 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:30:42.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:30:42.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:30:42.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:30:42.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836495 2024-09-18T15:30:42.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836495 2024-09-18T15:30:42.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836495' 2024-09-18T15:30:42.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:30:42.855 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:30:42.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672970 2024-09-18T15:30:42.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672970 2024-09-18T15:30:42.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836495 1-42949672970' 2024-09-18T15:30:42.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:30:42.998 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836495 2024-09-18T15:30:42.998 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:30:43.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:30:43.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836495 2024-09-18T15:30:43.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:30:43.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836495 2024-09-18T15:30:43.004 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836495 2024-09-18T15:30:43.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-09-18T15:30:43.005 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:30:43.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836493 -lt 21474836495 2024-09-18T15:30:43.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:30:44.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:30:44.315 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:30:44.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836495 -lt 21474836495 2024-09-18T15:30:44.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:30:44.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672970 2024-09-18T15:30:44.642 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:30:44.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:30:44.645 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:30:44.645 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672970 2024-09-18T15:30:44.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672970 2024-09-18T15:30:44.647 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672970 2024-09-18T15:30:44.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-09-18T15:30:44.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:30:44.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672970 -lt 42949672970 2024-09-18T15:30:44.989 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:30:44.990 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:30:44.990 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:30:45.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T15:30:45.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:30:45.409 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:30:45.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:30:45.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:30:45.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:30:45.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:30:45.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:30:45.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T15:30:45.753 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:30:45.754 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:30:45.754 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:30:46.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T15:30:46.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:30:46.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:30:46.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:103: TEST_corrupt_and_repair_replicated: add_something td/osd-scrub-repair rbd 2024-09-18T15:30:46.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:30:46.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=rbd 2024-09-18T15:30:46.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=SOMETHING 2024-09-18T15:30:46.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:30:46.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:30:46.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:30:46.727 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:30:46.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:30:47.276 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:30:47.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:30:47.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:30:47.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-18T15:30:47.529 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:104: TEST_corrupt_and_repair_replicated: get_not_primary rbd SOMETHING 2024-09-18T15:30:47.529 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-09-18T15:30:47.529 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-09-18T15:30:47.529 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-09-18T15:30:47.529 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-09-18T15:30:47.530 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T15:30:47.530 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-09-18T15:30:47.530 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:30:47.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T15:30:47.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-09-18T15:30:47.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T15:30:48.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:104: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-09-18T15:30:48.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:225: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-18T15:30:48.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local poolname=rbd 2024-09-18T15:30:48.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local osd=0 2024-09-18T15:30:48.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-18T15:30:48.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:30:48.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:30:48.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:30:48.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:30:48.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-18T15:30:48.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:30:48.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:30:48.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:30:48.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:30:48.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:30:48.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:30:48.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:30:48.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:30:48.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:30:48.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:30:48.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:30:48.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-18T15:30:48.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:30:48.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:30:48.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:30:48.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:30:48.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:30:48.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-18T15:30:50.220 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-18T15:30:50.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-18T15:30:50.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:30:50.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:30:50.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:30:50.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:30:50.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:30:50.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:30:50.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:30:50.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:30:50.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:30:50.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:30:50.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:30:50.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:30:50.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:30:50.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:30:50.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:30:50.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:30:50.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:30:50.757 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:30:50.758 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:30:50.758 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:30:50.758 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:30:50.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:30:50.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:30:50.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:30:50.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:30:50.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:30:50.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:30:50.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:30:50.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:30:50.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:30:50.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:30:50.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:30:50.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:30:50.760 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:30:50.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:30:50.762 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:30:50.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:30:50.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:30:50.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:30:50.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:30:50.793 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:30:50.787+0000 7f918c46c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:30:50.795 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:30:50.787+0000 7f918c46c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:30:50.797 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:30:50.791+0000 7f918c46c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:30:51.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:30:51.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:30:51.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:30:51.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:30:51.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:30:51.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:51.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:30:51.078 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:30:51.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:51.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:30:51.355 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:30:51.347+0000 7f918c46c540 -1 Falling back to public interface 2024-09-18T15:30:51.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:52.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:52.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:52.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:30:52.383 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:30:52.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:52.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:30:52.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:53.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:53.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:53.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:30:53.700 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:30:53.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:53.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:30:54.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:54.520 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:30:54.515+0000 7f918c46c540 -1 osd.0 22 log_to_monitors true 2024-09-18T15:30:55.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:55.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:55.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:30:55.015 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:30:55.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:55.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:30:55.405 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 26 up_thru 12 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6802/319186,v1:127.0.0.1:6803/319186] [v2:127.0.0.1:6804/319186,v1:127.0.0.1:6805/319186] exists,up d8472018-4074-42c2-b75f-ab3e86a0690e 2024-09-18T15:30:55.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:30:55.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:30:55.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:30:55.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:30:55.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:30:55.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:30:55.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:30:55.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:30:55.406 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:30:55.407 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:30:55.407 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:30:55.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:30:55.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:30:55.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:30:55.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:30:55.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:30:55.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:30:55.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:30:55.622 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:30:56.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:30:56.026 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:30:56.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:30:56.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:30:56.027 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:30:56.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=111669149698 2024-09-18T15:30:56.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 111669149698 2024-09-18T15:30:56.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-111669149698' 2024-09-18T15:30:56.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:30:56.267 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:30:56.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672973 2024-09-18T15:30:56.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672973 2024-09-18T15:30:56.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-111669149698 1-42949672973' 2024-09-18T15:30:56.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:30:56.528 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-111669149698 2024-09-18T15:30:56.528 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:30:56.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:30:56.530 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-111669149698 2024-09-18T15:30:56.530 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:30:56.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=111669149698 2024-09-18T15:30:56.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-09-18T15:30:56.533 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 111669149698 2024-09-18T15:30:56.533 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:30:56.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 111669149698 2024-09-18T15:30:56.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:30:57.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:30:57.840 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:30:58.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 111669149698 -lt 111669149698 2024-09-18T15:30:58.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:30:58.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672973 2024-09-18T15:30:58.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:30:58.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:30:58.157 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672973 2024-09-18T15:30:58.157 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:30:58.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672973 2024-09-18T15:30:58.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-09-18T15:30:58.160 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 42949672973 2024-09-18T15:30:58.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:30:58.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672973 -lt 42949672973 2024-09-18T15:30:58.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:30:58.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:30:58.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:30:58.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T15:30:58.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:30:58.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:30:58.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:30:58.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:30:58.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:30:58.864 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:30:58.864 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:30:59.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T15:30:59.152 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:30:59.152 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:30:59.152 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:30:59.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T15:30:59.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:30:59.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:30:59.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-09-18T15:30:59.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-09-18T15:30:59.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T15:30:59.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:30:59.513 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-09-18T15:30:59.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: local pg=1.3 2024-09-18T15:30:59.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: repair 1.3 2024-09-18T15:30:59.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=1.3 2024-09-18T15:30:59.839 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 1.3 2024-09-18T15:30:59.840 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T15:30:59.840 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:30:59.840 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:30:59.840 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T15:31:00.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T15:30:38.769699+0000 2024-09-18T15:31:00.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 1.3 2024-09-18T15:31:00.481 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 1.3 on osd.1 to repair 2024-09-18T15:31:00.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 1.3 2024-09-18T15:30:38.769699+0000 2024-09-18T15:31:00.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.3 2024-09-18T15:31:00.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:30:38.769699+0000 2024-09-18T15:31:00.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:31:00.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:31:00.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:31:00.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-18T15:31:00.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T15:31:00.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:31:00.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T15:31:00.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:31:00.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:30:38.769699+0000 '>' 2024-09-18T15:30:38.769699+0000 2024-09-18T15:31:00.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:31:01.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:31:01.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:31:01.798 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-18T15:31:01.798 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T15:31:01.798 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:31:01.799 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:31:01.799 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T15:31:02.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:30:38.769699+0000 '>' 2024-09-18T15:30:38.769699+0000 2024-09-18T15:31:02.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:31:03.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:31:03.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:31:03.119 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-18T15:31:03.119 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T15:31:03.119 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:31:03.119 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:31:03.119 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T15:31:03.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:30:38.769699+0000 '>' 2024-09-18T15:30:38.769699+0000 2024-09-18T15:31:03.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:31:04.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:31:04.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:31:04.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-18T15:31:04.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T15:31:04.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:31:04.436 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:31:04.436 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T15:31:04.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:31:02.677246+0000 '>' 2024-09-18T15:30:38.769699+0000 2024-09-18T15:31:04.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:31:04.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:241: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:31:04.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:31:04.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:31:04.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:31:04.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:31:04.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:31:04.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:31:04.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:31:04.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:31:04.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:31:04.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:31:04.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:31:04.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:31:04.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:31:04.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:31:04.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:31:05.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:31:05.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:31:05.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:31:05.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:31:05.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:31:05.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:31:05.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:31:05.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-18T15:31:06.771 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-18T15:31:06.771 INFO:tasks.workunit.client.0.smithi046.stdout:_ 2024-09-18T15:31:06.771 INFO:tasks.workunit.client.0.smithi046.stdout:snapset 2024-09-18T15:31:07.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:31:07.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:31:07.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:31:07.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:31:07.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:31:07.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:31:07.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:31:07.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:31:07.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:31:07.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:31:07.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:31:07.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:31:07.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:31:07.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:31:07.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:31:07.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:31:07.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:31:07.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:31:07.311 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:31:07.311 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:31:07.311 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:31:07.311 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:31:07.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:31:07.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:31:07.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:31:07.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:31:07.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:31:07.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:31:07.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:31:07.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:31:07.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:31:07.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:31:07.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:31:07.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:31:07.312 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:31:07.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:31:07.313 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:31:07.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:31:07.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:31:07.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:31:07.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:31:07.345 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:07.339+0000 7f86a37a3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:07.348 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:07.343+0000 7f86a37a3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:07.353 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:07.343+0000 7f86a37a3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:07.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:31:07.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:31:07.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:31:07.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:31:07.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:31:07.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:07.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:31:07.624 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:31:07.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:07.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:31:07.903 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:07.895+0000 7f86a37a3540 -1 Falling back to public interface 2024-09-18T15:31:07.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:08.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:08.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:08.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:31:08.998 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:31:08.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:08.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:31:09.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:10.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:10.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:10.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:31:10.310 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:31:10.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:10.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:31:10.644 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:10.635+0000 7f86a37a3540 -1 osd.1 28 log_to_monitors true 2024-09-18T15:31:10.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:11.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:11.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:11.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:31:11.840 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:31:11.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:11.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:31:12.238 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 32 up_thru 32 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/320664,v1:127.0.0.1:6811/320664] [v2:127.0.0.1:6812/320664,v1:127.0.0.1:6813/320664] exists,up 5775dd03-c507-469b-9a6a-e2fe091ddc66 2024-09-18T15:31:12.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:31:12.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:31:12.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:31:12.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:31:12.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:31:12.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:31:12.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:31:12.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:31:12.239 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:31:12.240 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:31:12.240 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:31:12.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:31:12.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:31:12.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:31:12.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:31:12.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:31:12.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:31:12.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:31:12.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:31:12.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:31:12.811 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:31:12.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:31:12.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:31:12.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:31:12.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=111669149702 2024-09-18T15:31:12.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 111669149702 2024-09-18T15:31:12.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-111669149702' 2024-09-18T15:31:12.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:31:12.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:31:13.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953474 2024-09-18T15:31:13.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953474 2024-09-18T15:31:13.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-111669149702 1-137438953474' 2024-09-18T15:31:13.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:31:13.134 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-111669149702 2024-09-18T15:31:13.135 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:31:13.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:31:13.137 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-111669149702 2024-09-18T15:31:13.138 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:31:13.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=111669149702 2024-09-18T15:31:13.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 111669149702' 2024-09-18T15:31:13.140 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 111669149702 2024-09-18T15:31:13.141 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:31:13.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 111669149701 -lt 111669149702 2024-09-18T15:31:13.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:31:14.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:31:14.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:31:14.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 111669149702 -lt 111669149702 2024-09-18T15:31:14.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:31:14.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-137438953474 2024-09-18T15:31:14.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:31:14.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:31:14.768 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-137438953474 2024-09-18T15:31:14.768 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:31:14.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953474 2024-09-18T15:31:14.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-09-18T15:31:14.770 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 137438953474 2024-09-18T15:31:14.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:31:15.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953474 -lt 137438953474 2024-09-18T15:31:15.124 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:31:15.124 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:31:15.125 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:31:15.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T15:31:15.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:31:15.615 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:31:15.615 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:31:15.615 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:31:15.616 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:31:15.616 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:31:15.616 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:31:15.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T15:31:15.939 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:31:15.939 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:31:15.940 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:31:16.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T15:31:16.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:31:16.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:31:16.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-09-18T15:31:16.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-18T15:31:16.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:106: TEST_corrupt_and_repair_replicated: get_primary rbd SOMETHING 2024-09-18T15:31:16.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-09-18T15:31:16.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T15:31:16.359 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-09-18T15:31:16.359 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:31:16.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:106: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 1 2024-09-18T15:31:16.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:225: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-18T15:31:16.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local poolname=rbd 2024-09-18T15:31:16.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local osd=1 2024-09-18T15:31:16.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:31:16.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:31:16.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:31:16.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:31:16.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:31:16.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:31:16.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:31:16.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:31:16.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:31:16.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:31:16.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:31:16.671 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:31:16.671 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:31:16.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:31:16.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:31:16.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:31:16.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:31:16.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-18T15:31:16.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:31:16.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:31:16.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:31:16.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:31:16.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:31:16.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-18T15:31:18.836 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-18T15:31:19.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:31:19.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:31:19.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:31:19.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:31:19.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:31:19.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:31:19.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:31:19.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:31:19.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:31:19.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:31:19.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:31:19.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:31:19.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:31:19.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:31:19.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:31:19.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:31:19.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:31:19.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:31:19.373 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:31:19.373 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:31:19.373 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:31:19.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:31:19.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:31:19.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:31:19.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:31:19.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:31:19.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:31:19.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:31:19.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:31:19.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:31:19.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:31:19.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:31:19.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:31:19.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:31:19.377 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:31:19.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:31:19.378 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:31:19.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:31:19.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:31:19.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:31:19.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:31:19.416 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:19.407+0000 7f13313d1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:19.420 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:19.411+0000 7f13313d1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:19.424 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:19.415+0000 7f13313d1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:19.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:31:19.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:31:19.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:31:19.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:31:19.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:31:19.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:19.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:31:19.692 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:31:19.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:19.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:31:19.726 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:19.719+0000 7f13313d1540 -1 Falling back to public interface 2024-09-18T15:31:20.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:21.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:21.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:21.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:31:21.041 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:31:21.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:21.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:31:21.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:22.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:22.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:22.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:31:22.361 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:31:22.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:22.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:31:22.539 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:22.531+0000 7f13313d1540 -1 osd.1 34 log_to_monitors true 2024-09-18T15:31:22.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:23.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:23.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:23.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:31:23.801 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:31:23.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:23.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:31:24.120 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 38 up_thru 38 down_at 35 last_clean_interval [32,34) [v2:127.0.0.1:6810/322012,v1:127.0.0.1:6811/322012] [v2:127.0.0.1:6812/322012,v1:127.0.0.1:6813/322012] exists,up 5775dd03-c507-469b-9a6a-e2fe091ddc66 2024-09-18T15:31:24.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:31:24.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:31:24.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:31:24.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:31:24.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:31:24.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:31:24.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:31:24.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:31:24.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:31:24.123 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:31:24.123 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:31:24.124 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:31:24.124 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:31:24.124 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:31:24.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:31:24.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:31:24.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:31:24.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:31:24.350 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:31:24.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:31:24.670 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:31:24.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:31:24.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:31:24.671 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:31:24.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=111669149705 2024-09-18T15:31:24.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 111669149705 2024-09-18T15:31:24.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-111669149705' 2024-09-18T15:31:24.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:31:24.804 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:31:24.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=163208757250 2024-09-18T15:31:24.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 163208757250 2024-09-18T15:31:24.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-111669149705 1-163208757250' 2024-09-18T15:31:24.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:31:24.944 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-111669149705 2024-09-18T15:31:24.945 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:31:24.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:31:24.948 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-111669149705 2024-09-18T15:31:24.948 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:31:24.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=111669149705 2024-09-18T15:31:24.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 111669149705' 2024-09-18T15:31:24.953 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 111669149705 2024-09-18T15:31:24.953 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:31:25.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 111669149704 -lt 111669149705 2024-09-18T15:31:25.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:31:26.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:31:26.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:31:26.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 111669149706 -lt 111669149705 2024-09-18T15:31:26.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:31:26.619 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-163208757250 2024-09-18T15:31:26.619 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:31:26.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:31:26.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-163208757250 2024-09-18T15:31:26.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:31:26.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=163208757250 2024-09-18T15:31:26.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 163208757250' 2024-09-18T15:31:26.625 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 163208757250 2024-09-18T15:31:26.626 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:31:26.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 163208757250 -lt 163208757250 2024-09-18T15:31:26.973 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:31:26.974 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:31:26.974 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:31:27.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T15:31:27.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:31:27.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:31:27.384 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:31:27.384 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:31:27.384 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:31:27.384 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:31:27.384 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:31:27.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T15:31:27.690 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:31:27.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:31:27.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:31:28.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T15:31:28.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:31:28.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:31:28.051 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-09-18T15:31:28.052 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-09-18T15:31:28.052 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T15:31:28.052 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-09-18T15:31:28.052 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:31:28.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:236: corrupt_and_repair_one: local pg=1.3 2024-09-18T15:31:28.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: repair 1.3 2024-09-18T15:31:28.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=1.3 2024-09-18T15:31:28.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 1.3 2024-09-18T15:31:28.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T15:31:28.460 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:31:28.460 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:31:28.460 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T15:31:28.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T15:31:02.677246+0000 2024-09-18T15:31:28.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 1.3 2024-09-18T15:31:29.066 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 1.3 on osd.1 to repair 2024-09-18T15:31:29.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 1.3 2024-09-18T15:31:02.677246+0000 2024-09-18T15:31:29.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.3 2024-09-18T15:31:29.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:31:02.677246+0000 2024-09-18T15:31:29.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:31:29.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:31:29.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:31:29.076 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-18T15:31:29.076 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T15:31:29.076 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:31:29.077 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:31:29.077 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T15:31:29.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:31:02.677246+0000 '>' 2024-09-18T15:31:02.677246+0000 2024-09-18T15:31:29.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:31:30.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:31:30.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:31:30.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-18T15:31:30.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T15:31:30.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:31:30.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:31:30.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T15:31:30.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:31:02.677246+0000 '>' 2024-09-18T15:31:02.677246+0000 2024-09-18T15:31:30.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:31:31.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:31:31.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:31:31.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-18T15:31:31.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T15:31:31.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:31:31.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:31:31.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T15:31:32.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:31:02.677246+0000 '>' 2024-09-18T15:31:02.677246+0000 2024-09-18T15:31:32.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:31:33.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:31:33.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:31:33.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-18T15:31:33.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T15:31:33.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:31:33.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:31:33.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T15:31:33.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:31:02.677246+0000 '>' 2024-09-18T15:31:02.677246+0000 2024-09-18T15:31:33.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:31:34.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:31:34.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:31:34.343 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-18T15:31:34.343 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T15:31:34.343 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:31:34.343 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:31:34.344 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T15:31:34.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:31:31.537495+0000 '>' 2024-09-18T15:31:02.677246+0000 2024-09-18T15:31:34.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:31:34.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:241: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:31:34.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:31:34.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:31:34.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:31:34.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:31:34.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:31:34.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:31:34.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:31:34.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:31:34.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:31:34.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:31:34.659 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:31:34.659 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:31:34.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:31:34.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:31:34.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:31:34.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:31:34.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-18T15:31:34.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:31:34.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:31:34.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:31:34.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:31:34.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:31:34.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-18T15:31:36.667 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-18T15:31:36.668 INFO:tasks.workunit.client.0.smithi046.stdout:_ 2024-09-18T15:31:36.668 INFO:tasks.workunit.client.0.smithi046.stdout:snapset 2024-09-18T15:31:37.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:31:37.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:31:37.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:31:37.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:31:37.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:31:37.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:31:37.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:31:37.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:31:37.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:31:37.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:31:37.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:31:37.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:31:37.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:31:37.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:31:37.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:31:37.209 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:31:37.209 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:31:37.209 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:31:37.209 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:31:37.209 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:31:37.209 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:31:37.209 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:31:37.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:31:37.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:31:37.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:31:37.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:31:37.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:31:37.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:31:37.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:31:37.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:31:37.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:31:37.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:31:37.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:31:37.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:31:37.212 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:31:37.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:31:37.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:31:37.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:31:37.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:31:37.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:31:37.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:31:37.245 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:37.239+0000 7f4dcd910540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:37.248 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:37.239+0000 7f4dcd910540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:37.249 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:37.243+0000 7f4dcd910540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:37.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:31:37.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:31:37.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:31:37.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:31:37.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:31:37.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:37.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:31:37.521 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:31:37.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:37.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:31:37.790 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:37.783+0000 7f4dcd910540 -1 Falling back to public interface 2024-09-18T15:31:37.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:38.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:38.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:38.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:31:38.903 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:31:38.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:38.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:31:39.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:40.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:40.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:40.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:31:40.217 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:31:40.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:40.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:31:40.475 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:40.467+0000 7f4dcd910540 -1 osd.1 39 log_to_monitors true 2024-09-18T15:31:40.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:41.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:41.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:41.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:31:41.581 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:31:41.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:41.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:31:42.205 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 40 last_clean_interval [38,39) [v2:127.0.0.1:6810/323516,v1:127.0.0.1:6811/323516] [v2:127.0.0.1:6812/323516,v1:127.0.0.1:6813/323516] exists,up 5775dd03-c507-469b-9a6a-e2fe091ddc66 2024-09-18T15:31:42.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:31:42.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:31:42.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:31:42.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:31:42.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:31:42.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:31:42.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:31:42.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:31:42.206 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:31:42.206 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:31:42.206 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:31:42.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:31:42.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:31:42.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:31:42.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:31:42.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:31:42.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:31:42.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:31:42.430 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:31:42.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:31:42.747 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:31:42.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:31:42.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:31:42.747 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:31:42.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=111669149710 2024-09-18T15:31:42.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 111669149710 2024-09-18T15:31:42.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-111669149710' 2024-09-18T15:31:42.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:31:42.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:31:43.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=184683593730 2024-09-18T15:31:43.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 184683593730 2024-09-18T15:31:43.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-111669149710 1-184683593730' 2024-09-18T15:31:43.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:31:43.084 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-111669149710 2024-09-18T15:31:43.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:31:43.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:31:43.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-111669149710 2024-09-18T15:31:43.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:31:43.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=111669149710 2024-09-18T15:31:43.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 111669149710' 2024-09-18T15:31:43.085 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 111669149710 2024-09-18T15:31:43.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:31:43.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 111669149709 -lt 111669149710 2024-09-18T15:31:43.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:31:44.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:31:44.388 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:31:44.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 111669149710 -lt 111669149710 2024-09-18T15:31:44.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:31:44.704 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-184683593730 2024-09-18T15:31:44.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:31:44.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:31:44.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-184683593730 2024-09-18T15:31:44.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:31:44.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=184683593730 2024-09-18T15:31:44.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 184683593730' 2024-09-18T15:31:44.711 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 184683593730 2024-09-18T15:31:44.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:31:45.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 184683593730 -lt 184683593730 2024-09-18T15:31:45.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:31:45.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:31:45.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:31:45.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T15:31:45.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:31:45.427 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:31:45.427 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:31:45.427 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:31:45.427 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:31:45.427 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:31:45.427 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:31:45.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T15:31:45.739 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:31:45.740 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:31:45.740 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:31:46.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T15:31:46.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:31:46.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:31:46.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-09-18T15:31:46.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-18T15:31:46.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T15:31:46.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:31:46.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:31:46.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:31:46.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:31:46.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:31:46.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:31:46.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:31:46.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:31:46.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:31:46.301 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:31:46.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:31:46.303 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:31:46.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:31:46.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:31:46.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:31:46.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:31:46.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:31:46.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:31:46.310 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:31:46.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:31:46.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:31:46.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:31:46.332 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:31:46.332 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:31:46.332 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:31:46.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:31:46.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:31:46.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:31:46.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T15:31:46.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T15:31:46.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T15:31:46.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T15:31:46.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:31:46.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:31:46.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:31:46.335 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:31:46.335 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:31:46.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:31:46.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:31:46.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:31:46.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:31:46.342 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:31:46.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:31:46.344 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:31:46.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:31:46.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:31:46.347 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:31:46.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:31:46.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:31:46.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:31:46.350 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:31:46.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:31:46.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:31:46.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:31:46.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:31:46.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:31:46.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:31:46.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:31:46.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:31:46.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:31:46.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T15:31:46.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:31:46.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:31:46.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:31:46.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T15:31:46.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:31:46.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:31:46.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:31:46.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T15:31:46.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_corrupt_scrub_erasure_appends td/osd-scrub-repair 2024-09-18T15:31:46.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5767: TEST_corrupt_scrub_erasure_appends: corrupt_scrub_erasure td/osd-scrub-repair false 2024-09-18T15:31:46.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-09-18T15:31:46.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: local allow_overwrites=false 2024-09-18T15:31:46.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3615: corrupt_scrub_erasure: local poolname=ecpool 2024-09-18T15:31:46.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local total_objs=7 2024-09-18T15:31:46.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-09-18T15:31:46.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T15:31:46.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:31:46.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:31:46.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:31:46.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T15:31:46.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-18T15:31:46.815 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:31:46.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:31:46.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:31:46.816 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:31:46.817 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:31:46.817 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:31:46.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:31:46.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T15:31:47.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:31:47.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:31:47.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:31:47.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:31:47.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:31:47.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:31:47.097 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:31:47.097 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:31:47.097 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:31:47.098 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:31:47.098 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:31:47.098 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:31:47.099 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:31:47.099 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:31:47.099 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T15:31:47.198 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:31:47.198 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:31:47.198 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:31:47.198 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:31:47.199 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:31:47.199 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:31:47.199 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:31:47.199 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:31:47.200 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:31:47.200 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:31:47.200 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:31:47.200 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:31:47.201 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:31:47.201 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T15:31:47.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3619: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-09-18T15:31:47.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T15:31:47.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:31:47.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:31:47.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:31:47.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T15:31:47.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:31:48.157 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:31:48.157 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:31:48.157 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:31:48.157 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:31:48.158 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:31:48.158 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:31:48.158 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:31:48.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:31:48.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:31:48.199 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3620: corrupt_scrub_erasure: seq 0 2 2024-09-18T15:31:48.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3620: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-18T15:31:48.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: '[' false = true ']' 2024-09-18T15:31:48.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3624: corrupt_scrub_erasure: run_osd_filestore td/osd-scrub-repair 0 2024-09-18T15:31:48.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T15:31:48.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T15:31:48.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=0 2024-09-18T15:31:48.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T15:31:48.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:31:48.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:31:48.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:31:48.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:31:48.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:31:48.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:31:48.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:31:48.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T15:31:48.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T15:31:48.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:31:48.203 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T15:31:48.203 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:31:48.203 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:31:48.205 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:31:48.205 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:31:48.205 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:31:48.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:31:48.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:31:48.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T15:31:48.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T15:31:48.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T15:31:48.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:31:48.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:31:48.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:31:48.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:31:48.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:31:48.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T15:31:48.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T15:31:48.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:31:48.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T15:31:48.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=9709534c-708f-4053-bc68-1443befb5261 2024-09-18T15:31:48.209 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 9709534c-708f-4053-bc68-1443befb5261 2024-09-18T15:31:48.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd1 9709534c-708f-4053-bc68-1443befb5261' 2024-09-18T15:31:48.209 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T15:31:48.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQBk8upmo9s4DRAA9XZfUzHQlN9bQluRSEU4xQ== 2024-09-18T15:31:48.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQBk8upmo9s4DRAA9XZfUzHQlN9bQluRSEU4xQ=="}' 2024-09-18T15:31:48.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 9709534c-708f-4053-bc68-1443befb5261 -i td/osd-scrub-repair/0/new.json 2024-09-18T15:31:48.640 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:31:48.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/0/new.json 2024-09-18T15:31:48.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBk8upmo9s4DRAA9XZfUzHQlN9bQluRSEU4xQ== --osd-uuid 9709534c-708f-4053-bc68-1443befb5261 --osd-objectstore=filestore 2024-09-18T15:31:48.689 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:48.683+0000 7f2e6707b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:48.692 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:48.687+0000 7f2e6707b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:48.694 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:48.687+0000 7f2e6707b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:49.183 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:49.175+0000 7f2e6707b540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:31:49.671 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:49.663+0000 7f2e6707b540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:31:49.671 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:49.663+0000 7f2e6707b540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:31:49.671 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:49.663+0000 7f2e6707b540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:31:50.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T15:31:50.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T15:31:50.280 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T15:31:50.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd0 key to auth repository 2024-09-18T15:31:50.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:31:50.461 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: key for osd.0 not found in provided keyring 2024-09-18T15:31:50.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.0 2024-09-18T15:31:50.462 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T15:31:50.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:31:50.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T15:31:50.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T15:31:50.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T15:31:50.491 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:50.483+0000 7f44e7f83540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:50.493 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:50.487+0000 7f44e7f83540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:50.501 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:50.487+0000 7f44e7f83540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:50.501 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:50.487+0000 7f44e7f83540 -1 Falling back to public interface 2024-09-18T15:31:50.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 0 2024-09-18T15:31:50.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:31:50.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:31:50.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:31:50.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:31:50.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:50.672 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:31:50.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:31:50.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:50.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:31:50.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:51.124 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:51.119+0000 7f44e7f83540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:31:51.127 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:51.119+0000 7f44e7f83540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:31:51.127 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:51.119+0000 7f44e7f83540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:31:51.137 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:51.131+0000 7f44e7f83540 -1 osd.0 0 log_to_monitors true 2024-09-18T15:31:51.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:51.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:51.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:31:51.902 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:31:51.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:51.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:31:52.086 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/324761,v1:127.0.0.1:6801/324761] [v2:127.0.0.1:6802/324761,v1:127.0.0.1:6803/324761] exists,up 9709534c-708f-4053-bc68-1443befb5261 2024-09-18T15:31:52.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:31:52.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:31:52.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:31:52.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3620: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-18T15:31:52.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: '[' false = true ']' 2024-09-18T15:31:52.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3624: corrupt_scrub_erasure: run_osd_filestore td/osd-scrub-repair 1 2024-09-18T15:31:52.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T15:31:52.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T15:31:52.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=1 2024-09-18T15:31:52.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T15:31:52.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:31:52.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:31:52.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:31:52.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:31:52.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:31:52.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:31:52.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:31:52.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T15:31:52.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T15:31:52.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:31:52.090 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T15:31:52.090 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:31:52.090 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:31:52.091 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:31:52.091 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:31:52.091 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:31:52.092 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:31:52.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:31:52.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T15:31:52.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T15:31:52.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T15:31:52.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:31:52.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:31:52.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:31:52.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:31:52.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:31:52.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T15:31:52.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T15:31:52.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:31:52.093 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T15:31:52.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=a98783e8-5660-4fe2-8f10-19ac7db7df61 2024-09-18T15:31:52.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd1 a98783e8-5660-4fe2-8f10-19ac7db7df61' 2024-09-18T15:31:52.094 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 a98783e8-5660-4fe2-8f10-19ac7db7df61 2024-09-18T15:31:52.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T15:31:52.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQBo8upmUEN1BhAAUt9+ROqEntUiLcKVHENEdw== 2024-09-18T15:31:52.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQBo8upmUEN1BhAAUt9+ROqEntUiLcKVHENEdw=="}' 2024-09-18T15:31:52.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new a98783e8-5660-4fe2-8f10-19ac7db7df61 -i td/osd-scrub-repair/1/new.json 2024-09-18T15:31:52.478 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:31:52.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/1/new.json 2024-09-18T15:31:52.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBo8upmUEN1BhAAUt9+ROqEntUiLcKVHENEdw== --osd-uuid a98783e8-5660-4fe2-8f10-19ac7db7df61 --osd-objectstore=filestore 2024-09-18T15:31:52.519 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:52.511+0000 7f22a7c3a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:52.527 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:52.519+0000 7f22a7c3a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:52.529 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:52.523+0000 7f22a7c3a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:53.113 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:53.107+0000 7f22a7c3a540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:31:53.760 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:53.755+0000 7f22a7c3a540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:31:53.767 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:53.759+0000 7f22a7c3a540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:31:53.767 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:53.759+0000 7f22a7c3a540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:31:54.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-18T15:31:54.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T15:31:54.916 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T15:31:54.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd1 key to auth repository 2024-09-18T15:31:54.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:31:55.298 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T15:31:55.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.1 2024-09-18T15:31:55.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:31:55.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T15:31:55.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T15:31:55.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T15:31:55.329 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:55.323+0000 7fd80881a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:55.332 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:55.327+0000 7fd80881a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:55.340 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:55.327+0000 7fd80881a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:55.341 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:55.335+0000 7fd80881a540 -1 Falling back to public interface 2024-09-18T15:31:55.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 1 2024-09-18T15:31:55.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:31:55.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:31:55.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:31:55.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:31:55.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:55.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:31:55.750 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:31:55.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:55.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:31:56.003 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:55.995+0000 7fd80881a540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:31:56.019 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:56.011+0000 7fd80881a540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:31:56.019 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:56.011+0000 7fd80881a540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:31:56.044 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:56.035+0000 7fd80881a540 -1 osd.1 0 log_to_monitors true 2024-09-18T15:31:56.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:57.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:57.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:57.075 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:31:57.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:31:57.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:57.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:31:57.435 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/325144,v1:127.0.0.1:6811/325144] [v2:127.0.0.1:6812/325144,v1:127.0.0.1:6813/325144] exists,up a98783e8-5660-4fe2-8f10-19ac7db7df61 2024-09-18T15:31:57.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:31:57.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:31:57.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:31:57.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3620: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-18T15:31:57.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: '[' false = true ']' 2024-09-18T15:31:57.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3624: corrupt_scrub_erasure: run_osd_filestore td/osd-scrub-repair 2 2024-09-18T15:31:57.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T15:31:57.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T15:31:57.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=2 2024-09-18T15:31:57.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T15:31:57.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:31:57.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:31:57.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:31:57.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:31:57.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:31:57.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T15:31:57.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T15:31:57.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T15:31:57.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T15:31:57.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:31:57.438 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T15:31:57.438 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:31:57.438 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:31:57.440 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:31:57.440 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:31:57.440 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:31:57.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:31:57.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:31:57.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T15:31:57.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T15:31:57.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T15:31:57.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:31:57.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:31:57.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:31:57.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:31:57.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:31:57.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T15:31:57.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T15:31:57.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/2 2024-09-18T15:31:57.441 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T15:31:57.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=3c30fbf0-6eac-4a83-90f1-b24860579ebc 2024-09-18T15:31:57.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd1 3c30fbf0-6eac-4a83-90f1-b24860579ebc' 2024-09-18T15:31:57.443 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 3c30fbf0-6eac-4a83-90f1-b24860579ebc 2024-09-18T15:31:57.444 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T15:31:57.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQBt8upme1N5GxAAvZLqFjdEMt+2XS9MeYijLw== 2024-09-18T15:31:57.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQBt8upme1N5GxAAvZLqFjdEMt+2XS9MeYijLw=="}' 2024-09-18T15:31:57.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 3c30fbf0-6eac-4a83-90f1-b24860579ebc -i td/osd-scrub-repair/2/new.json 2024-09-18T15:31:57.887 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:31:57.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/2/new.json 2024-09-18T15:31:57.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBt8upme1N5GxAAvZLqFjdEMt+2XS9MeYijLw== --osd-uuid 3c30fbf0-6eac-4a83-90f1-b24860579ebc --osd-objectstore=filestore 2024-09-18T15:31:57.936 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:57.927+0000 7f51221cd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:57.939 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:57.931+0000 7f51221cd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:57.941 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:57.935+0000 7f51221cd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:58.668 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:58.663+0000 7f51221cd540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:31:59.388 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:59.379+0000 7f51221cd540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:31:59.388 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:59.379+0000 7f51221cd540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:31:59.388 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:31:59.379+0000 7f51221cd540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:32:00.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-18T15:32:00.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T15:32:00.186 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T15:32:00.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd2 key to auth repository 2024-09-18T15:32:00.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:32:00.559 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: key for osd.2 not found in provided keyring 2024-09-18T15:32:00.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.2 2024-09-18T15:32:00.562 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T15:32:00.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T15:32:00.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T15:32:00.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:32:00.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T15:32:00.592 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:00.583+0000 7f5bf1caf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:32:00.595 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:00.587+0000 7f5bf1caf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:32:00.596 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:00.591+0000 7f5bf1caf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:32:00.598 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:00.591+0000 7f5bf1caf540 -1 Falling back to public interface 2024-09-18T15:32:00.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 2 2024-09-18T15:32:00.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:32:00.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:32:00.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:32:00.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:32:00.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:32:00.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:32:00.988 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:32:00.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:32:00.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:32:01.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:32:01.322 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:01.315+0000 7f5bf1caf540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:32:01.322 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:01.315+0000 7f5bf1caf540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:32:01.322 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:01.315+0000 7f5bf1caf540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:32:01.330 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:01.323+0000 7f5bf1caf540 -1 osd.2 0 log_to_monitors true 2024-09-18T15:32:02.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:32:02.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:32:02.316 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:32:02.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:32:02.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:32:02.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:32:02.721 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/325427,v1:127.0.0.1:6819/325427] [v2:127.0.0.1:6820/325427,v1:127.0.0.1:6821/325427] exists,up 3c30fbf0-6eac-4a83-90f1-b24860579ebc 2024-09-18T15:32:02.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:32:02.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:32:02.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:32:02.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3627: corrupt_scrub_erasure: create_rbd_pool 2024-09-18T15:32:02.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T15:32:03.054 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' does not exist 2024-09-18T15:32:03.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T15:32:03.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T15:32:03.707 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' created 2024-09-18T15:32:03.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:32:04.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T15:32:07.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: create_pool foo 1 2024-09-18T15:32:07.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-09-18T15:32:08.145 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'foo' created 2024-09-18T15:32:08.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:32:09.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3630: corrupt_scrub_erasure: create_ec_pool ecpool false k=2 m=1 stripe_unit=2K --force 2024-09-18T15:32:09.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: create_ec_pool: local pool_name=ecpool 2024-09-18T15:32:09.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: create_ec_pool: shift 2024-09-18T15:32:09.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: create_ec_pool: local allow_overwrites=false 2024-09-18T15:32:09.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: create_ec_pool: shift 2024-09-18T15:32:09.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-09-18T15:32:09.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-18T15:32:09.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-18T15:32:10.458 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'ecpool' created 2024-09-18T15:32:10.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:32:11.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: create_ec_pool: '[' false = true ']' 2024-09-18T15:32:11.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: create_ec_pool: wait_for_clean 2024-09-18T15:32:11.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:32:11.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:32:11.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:32:11.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:32:11.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:32:11.475 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:32:11.475 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:32:11.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:32:11.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:32:11.476 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:32:11.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:32:11.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:32:11.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:32:11.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:32:11.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:32:12.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:32:12.022 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:32:12.022 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:32:12.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:32:12.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:32:12.023 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:32:12.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836485 2024-09-18T15:32:12.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836485 2024-09-18T15:32:12.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836485' 2024-09-18T15:32:12.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:32:12.196 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:32:12.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672964 2024-09-18T15:32:12.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672964 2024-09-18T15:32:12.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836485 1-42949672964' 2024-09-18T15:32:12.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:32:12.511 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:32:12.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509443 2024-09-18T15:32:12.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509443 2024-09-18T15:32:12.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836485 1-42949672964 2-64424509443' 2024-09-18T15:32:12.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:32:12.682 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836485 2024-09-18T15:32:12.682 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:32:12.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:32:12.685 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836485 2024-09-18T15:32:12.685 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:32:12.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836485 2024-09-18T15:32:12.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-09-18T15:32:12.688 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836485 2024-09-18T15:32:12.688 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:32:13.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836484 -lt 21474836485 2024-09-18T15:32:13.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:32:14.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:32:14.084 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:32:14.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836485 -lt 21474836485 2024-09-18T15:32:14.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:32:14.396 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672964 2024-09-18T15:32:14.396 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:32:14.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:32:14.400 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672964 2024-09-18T15:32:14.401 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:32:14.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672964 2024-09-18T15:32:14.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-09-18T15:32:14.403 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672964 2024-09-18T15:32:14.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:32:14.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672965 -lt 42949672964 2024-09-18T15:32:14.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:32:14.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509443 2024-09-18T15:32:14.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:32:14.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:32:14.725 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509443 2024-09-18T15:32:14.725 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:32:14.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509443 2024-09-18T15:32:14.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2024-09-18T15:32:14.728 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509443 2024-09-18T15:32:14.728 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:32:15.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509444 -lt 64424509443 2024-09-18T15:32:15.084 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:32:15.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:15.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:32:15.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:32:15.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:32:15.499 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:32:15.499 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:32:15.499 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:32:15.499 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:32:15.499 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:32:15.499 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:32:15.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:32:15.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:32:15.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:15.866 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:32:16.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:32:16.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:32:16.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:32:16.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: create_ec_pool: return 0 2024-09-18T15:32:16.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3631: corrupt_scrub_erasure: wait_for_clean 2024-09-18T15:32:16.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:32:16.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:32:16.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:32:16.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:32:16.230 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:32:16.230 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:32:16.230 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:32:16.230 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:32:16.230 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:32:16.230 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:32:16.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:32:16.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:32:16.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:32:16.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:32:16.454 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:32:16.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:32:16.755 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:32:16.756 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:32:16.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:32:16.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:32:16.756 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:32:16.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836487 2024-09-18T15:32:16.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836487 2024-09-18T15:32:16.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836487' 2024-09-18T15:32:16.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:32:16.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:32:17.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672966 2024-09-18T15:32:17.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672966 2024-09-18T15:32:17.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836487 1-42949672966' 2024-09-18T15:32:17.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:32:17.063 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:32:17.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509445 2024-09-18T15:32:17.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509445 2024-09-18T15:32:17.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836487 1-42949672966 2-64424509445' 2024-09-18T15:32:17.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:32:17.199 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836487 2024-09-18T15:32:17.200 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:32:17.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:32:17.203 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836487 2024-09-18T15:32:17.203 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:32:17.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836487 2024-09-18T15:32:17.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-09-18T15:32:17.205 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836487 2024-09-18T15:32:17.206 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:32:17.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836486 -lt 21474836487 2024-09-18T15:32:17.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:32:18.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:32:18.524 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:32:18.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836487 -lt 21474836487 2024-09-18T15:32:18.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:32:18.840 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672966 2024-09-18T15:32:18.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:32:18.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:32:18.844 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672966 2024-09-18T15:32:18.845 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:32:18.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672966 2024-09-18T15:32:18.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-09-18T15:32:18.848 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672966 2024-09-18T15:32:18.848 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:32:19.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672966 -lt 42949672966 2024-09-18T15:32:19.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:32:19.202 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509445 2024-09-18T15:32:19.202 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:32:19.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:32:19.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509445 2024-09-18T15:32:19.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:32:19.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509445 2024-09-18T15:32:19.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-09-18T15:32:19.208 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509445 2024-09-18T15:32:19.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:32:19.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509445 -lt 64424509445 2024-09-18T15:32:19.534 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:32:19.534 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:19.534 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:32:19.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:32:19.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:32:19.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:32:19.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:32:19.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:32:19.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:32:19.904 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:32:19.904 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:32:20.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:32:20.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:32:20.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:20.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:32:20.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:32:20.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:32:20.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:32:20.634 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: seq 1 7 2024-09-18T15:32:20.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-18T15:32:20.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objname=EOBJ1 2024-09-18T15:32:20.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-09-18T15:32:20.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:32:20.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2024-09-18T15:32:20.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ1 2024-09-18T15:32:20.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:32:20.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:32:20.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:32:21.129 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:32:21.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:32:21.715 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:32:21.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:32:21.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:32:21.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:32:21.880 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: expr 1 % 2 2024-09-18T15:32:21.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: local osd=1 2024-09-18T15:32:21.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: case $i in 2024-09-18T15:32:21.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-09-18T15:32:21.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: echo UVWXYZZZ 2024-09-18T15:32:21.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:32:21.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:32:21.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:32:21.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:32:21.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:32:21.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:32:21.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:32:21.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:32:21.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:32:21.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:32:21.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:32:21.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:32:21.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:32:21.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:32:21.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:32:21.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:32:22.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:32:22.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:32:22.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:32:22.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:32:22.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:32:22.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:32:22.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:32:22.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:32:23.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:32:23.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:32:23.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:32:23.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:32:23.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:32:23.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:32:23.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:32:23.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:32:23.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:32:23.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:32:23.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:32:23.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:32:23.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:32:23.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:32:23.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:32:23.447 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:32:23.447 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:32:23.448 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:32:23.448 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:32:23.448 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:32:23.448 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:32:23.448 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:32:23.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:32:23.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:32:23.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:32:23.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:32:23.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:32:23.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:32:23.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:32:23.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:32:23.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:32:23.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:32:23.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:32:23.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:32:23.451 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:32:23.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:32:23.452 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:32:23.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:32:23.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:32:23.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:32:23.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:32:23.483 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:23.474+0000 7f4273fc2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:32:23.487 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:23.478+0000 7f4273fc2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:32:23.489 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:23.482+0000 7f4273fc2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:32:23.493 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:23.486+0000 7f4273fc2540 -1 Falling back to public interface 2024-09-18T15:32:23.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:32:23.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:32:23.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:32:23.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:32:23.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:32:23.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:32:23.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:32:23.773 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:32:23.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:32:23.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:32:23.971 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:23.962+0000 7f4273fc2540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:32:23.986 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:23.978+0000 7f4273fc2540 -1 journal do_read_entry(405504): bad header magic 2024-09-18T15:32:23.986 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:23.978+0000 7f4273fc2540 -1 journal do_read_entry(405504): bad header magic 2024-09-18T15:32:24.010 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:24.002+0000 7f4273fc2540 -1 osd.1 36 log_to_monitors true 2024-09-18T15:32:24.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:32:25.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:32:25.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:32:25.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:32:25.147 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:32:25.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:32:25.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:32:25.457 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [10,36) [v2:127.0.0.1:6810/326622,v1:127.0.0.1:6811/326622] [v2:127.0.0.1:6812/326622,v1:127.0.0.1:6813/326622] exists,up a98783e8-5660-4fe2-8f10-19ac7db7df61 2024-09-18T15:32:25.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:32:25.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:32:25.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:32:25.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:32:25.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:32:25.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:32:25.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:32:25.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:32:25.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:32:25.459 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:32:25.459 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:32:25.460 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:32:25.460 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:32:25.460 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:32:25.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:32:25.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:32:25.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:32:25.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:32:25.681 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:32:25.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:32:25.998 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:32:25.999 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:32:25.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:32:25.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:32:25.999 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:32:26.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836490 2024-09-18T15:32:26.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836490 2024-09-18T15:32:26.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836490' 2024-09-18T15:32:26.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:32:26.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:32:26.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691842 2024-09-18T15:32:26.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691842 2024-09-18T15:32:26.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836490 1-171798691842' 2024-09-18T15:32:26.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:32:26.323 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:32:26.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509448 2024-09-18T15:32:26.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509448 2024-09-18T15:32:26.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836490 1-171798691842 2-64424509448' 2024-09-18T15:32:26.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:32:26.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836490 2024-09-18T15:32:26.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:32:26.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:32:26.465 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836490 2024-09-18T15:32:26.465 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:32:26.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836490 2024-09-18T15:32:26.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-09-18T15:32:26.467 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836490 2024-09-18T15:32:26.467 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:32:26.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836489 -lt 21474836490 2024-09-18T15:32:26.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:32:27.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:32:27.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:32:28.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836490 -lt 21474836490 2024-09-18T15:32:28.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:32:28.224 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-171798691842 2024-09-18T15:32:28.224 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:32:28.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:32:28.227 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-171798691842 2024-09-18T15:32:28.227 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:32:28.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691842 2024-09-18T15:32:28.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2024-09-18T15:32:28.230 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 171798691842 2024-09-18T15:32:28.230 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:32:28.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691842 -lt 171798691842 2024-09-18T15:32:28.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:32:28.583 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509448 2024-09-18T15:32:28.583 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:32:28.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:32:28.586 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509448 2024-09-18T15:32:28.587 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:32:28.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509448 2024-09-18T15:32:28.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-18T15:32:28.589 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509448 2024-09-18T15:32:28.590 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:32:28.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509448 -lt 64424509448 2024-09-18T15:32:28.936 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:32:28.936 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:32:28.937 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:29.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:32:29.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:32:29.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:32:29.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:32:29.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:32:29.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:32:29.305 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:32:29.305 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:32:29.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:32:29.600 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:32:29.601 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:29.601 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:32:29.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:32:29.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:32:29.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:32:29.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-18T15:32:29.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objname=EOBJ2 2024-09-18T15:32:29.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-09-18T15:32:29.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:32:29.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2024-09-18T15:32:29.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ2 2024-09-18T15:32:29.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:32:29.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:32:29.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:32:30.593 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:32:30.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:32:31.064 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:32:31.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:32:31.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:32:31.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:32:31.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: expr 2 % 2 2024-09-18T15:32:31.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: local osd=0 2024-09-18T15:32:31.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: case $i in 2024-09-18T15:32:31.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3649: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-09-18T15:32:31.257 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records in 2024-09-18T15:32:31.257 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records out 2024-09-18T15:32:31.257 INFO:tasks.workunit.client.0.smithi046.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 0.000246942 s, 8.3 MB/s 2024-09-18T15:32:31.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3650: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:32:31.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:32:31.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:32:31.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:32:31.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:32:31.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:32:31.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:32:31.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:32:31.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:32:31.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:32:31.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:32:31.260 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:32:31.260 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:32:31.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:32:31.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:32:31.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:32:31.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:32:31.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:32:31.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:32:31.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:32:31.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:32:31.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:32:31.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:32:31.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:32:32.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-18T15:32:32.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:32:32.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:32:32.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:32:32.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:32:32.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:32:32.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:32:32.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:32:32.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:32:32.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:32:32.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:32:32.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:32:32.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:32:32.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:32:32.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:32:32.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:32:32.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:32:32.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:32:32.757 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:32:32.757 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:32:32.757 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:32:32.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:32:32.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:32:32.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:32:32.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:32:32.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:32:32.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:32:32.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:32:32.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:32:32.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:32:32.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:32:32.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:32:32.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:32:32.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:32:32.760 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:32:32.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:32:32.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:32:32.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:32:32.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:32:32.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:32:32.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:32:32.790 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:32.782+0000 7fa6254cd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:32:32.794 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:32.786+0000 7fa6254cd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:32:32.795 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:32.786+0000 7fa6254cd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:32:32.799 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:32.790+0000 7fa6254cd540 -1 Falling back to public interface 2024-09-18T15:32:33.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:32:33.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:32:33.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:32:33.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:32:33.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:32:33.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:32:33.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:32:33.071 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:32:33.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:32:33.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:32:33.328 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:33.322+0000 7fa6254cd540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:32:33.329 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:33.322+0000 7fa6254cd540 -1 journal do_read_entry(671744): bad header magic 2024-09-18T15:32:33.329 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:33.322+0000 7fa6254cd540 -1 journal do_read_entry(671744): bad header magic 2024-09-18T15:32:33.341 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:33.334+0000 7fa6254cd540 -1 osd.0 43 log_to_monitors true 2024-09-18T15:32:33.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:32:34.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:32:34.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:32:34.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:32:34.388 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:32:34.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:32:34.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:32:34.796 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 47 up_thru 47 down_at 44 last_clean_interval [5,43) [v2:127.0.0.1:6800/327295,v1:127.0.0.1:6801/327295] [v2:127.0.0.1:6802/327295,v1:127.0.0.1:6803/327295] exists,up 9709534c-708f-4053-bc68-1443befb5261 2024-09-18T15:32:34.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:32:34.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:32:34.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:32:34.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:32:34.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:32:34.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:32:34.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:32:34.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:32:34.797 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:32:34.797 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:32:34.797 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:32:34.798 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:32:34.798 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:32:34.798 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:32:35.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:32:35.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:32:35.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:32:35.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:32:35.016 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:32:35.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:32:35.337 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:32:35.337 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:32:35.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:32:35.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:32:35.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:32:35.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=201863462914 2024-09-18T15:32:35.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 201863462914 2024-09-18T15:32:35.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-201863462914' 2024-09-18T15:32:35.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:32:35.518 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:32:35.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691845 2024-09-18T15:32:35.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691845 2024-09-18T15:32:35.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-201863462914 1-171798691845' 2024-09-18T15:32:35.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:32:35.702 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:32:35.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509451 2024-09-18T15:32:35.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509451 2024-09-18T15:32:35.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-201863462914 1-171798691845 2-64424509451' 2024-09-18T15:32:35.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:32:35.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-201863462914 2024-09-18T15:32:35.851 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:32:35.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:32:35.854 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-201863462914 2024-09-18T15:32:35.854 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:32:35.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=201863462914 2024-09-18T15:32:35.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 201863462914' 2024-09-18T15:32:35.856 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 201863462914 2024-09-18T15:32:35.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:32:36.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 201863462914 -lt 201863462914 2024-09-18T15:32:36.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:32:36.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-171798691845 2024-09-18T15:32:36.162 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:32:36.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:32:36.165 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-171798691845 2024-09-18T15:32:36.165 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:32:36.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691845 2024-09-18T15:32:36.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 171798691845' 2024-09-18T15:32:36.168 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 171798691845 2024-09-18T15:32:36.168 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:32:36.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691844 -lt 171798691845 2024-09-18T15:32:36.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:32:37.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:32:37.494 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:32:37.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691845 -lt 171798691845 2024-09-18T15:32:37.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:32:37.810 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509451 2024-09-18T15:32:37.810 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:32:37.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:32:37.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509451 2024-09-18T15:32:37.814 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:32:37.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509451 2024-09-18T15:32:37.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-18T15:32:37.816 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509451 2024-09-18T15:32:37.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:32:38.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509451 -lt 64424509451 2024-09-18T15:32:38.157 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:32:38.157 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:38.158 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:32:38.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:32:38.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:32:38.617 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:32:38.617 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:32:38.617 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:32:38.617 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:32:38.618 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:32:38.618 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:32:38.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:32:38.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:32:38.965 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:38.965 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:32:39.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:32:39.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:32:39.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:32:39.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-18T15:32:39.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objname=EOBJ3 2024-09-18T15:32:39.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-09-18T15:32:39.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:32:39.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2024-09-18T15:32:39.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ3 2024-09-18T15:32:39.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:32:39.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:32:39.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:32:39.911 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:32:39.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:32:40.321 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:32:40.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:32:40.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:32:40.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:32:40.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: expr 3 % 2 2024-09-18T15:32:40.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: local osd=1 2024-09-18T15:32:40.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: case $i in 2024-09-18T15:32:40.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3655: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-09-18T15:32:40.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:32:40.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:32:40.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:32:40.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:32:40.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-09-18T15:32:40.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:32:40.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:32:40.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:32:40.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:32:40.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:32:40.516 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:32:40.516 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:32:40.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:32:40.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:32:40.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:32:40.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:32:40.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-09-18T15:32:40.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:32:40.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:32:40.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:32:40.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:32:40.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:32:40.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-09-18T15:32:41.709 INFO:tasks.workunit.client.0.smithi046.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-09-18T15:32:42.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:32:42.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:32:42.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:32:42.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:32:42.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:32:42.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:32:42.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:32:42.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:32:42.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:32:42.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:32:42.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:32:42.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:32:42.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:32:42.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:32:42.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:32:42.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:32:42.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:32:42.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:32:42.292 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:32:42.292 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:32:42.292 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:32:42.292 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:32:42.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:32:42.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:32:42.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:32:42.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:32:42.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:32:42.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:32:42.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:32:42.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:32:42.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:32:42.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:32:42.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:32:42.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:32:42.294 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:32:42.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:32:42.296 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:32:42.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:32:42.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:32:42.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:32:42.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:32:42.328 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:42.318+0000 7fcceae45540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:32:42.331 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:42.322+0000 7fcceae45540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:32:42.333 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:42.326+0000 7fcceae45540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:32:42.335 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:42.326+0000 7fcceae45540 -1 Falling back to public interface 2024-09-18T15:32:42.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:32:42.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:32:42.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:32:42.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:32:42.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:32:42.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:32:42.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:32:42.657 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:32:42.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:32:42.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:32:42.839 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:42.830+0000 7fcceae45540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:32:42.845 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:42.838+0000 7fcceae45540 -1 journal do_read_entry(937984): bad header magic 2024-09-18T15:32:42.845 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:42.838+0000 7fcceae45540 -1 journal do_read_entry(937984): bad header magic 2024-09-18T15:32:42.858 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:42.850+0000 7fcceae45540 -1 osd.1 51 log_to_monitors true 2024-09-18T15:32:43.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:32:44.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:32:44.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:32:44.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:32:44.044 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:32:44.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:32:44.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:32:44.446 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 56 up_thru 56 down_at 52 last_clean_interval [40,51) [v2:127.0.0.1:6810/327968,v1:127.0.0.1:6811/327968] [v2:127.0.0.1:6812/327968,v1:127.0.0.1:6813/327968] exists,up a98783e8-5660-4fe2-8f10-19ac7db7df61 2024-09-18T15:32:44.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:32:44.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:32:44.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:32:44.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:32:44.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:32:44.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:32:44.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:32:44.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:32:44.447 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:32:44.448 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:32:44.448 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:32:44.449 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:32:44.449 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:32:44.449 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:32:44.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:32:44.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:32:44.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:32:44.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:32:44.678 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:32:45.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:32:45.020 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:32:45.020 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:32:45.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:32:45.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:32:45.021 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:32:45.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=201863462917 2024-09-18T15:32:45.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 201863462917 2024-09-18T15:32:45.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-201863462917' 2024-09-18T15:32:45.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:32:45.168 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:32:45.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=240518168578 2024-09-18T15:32:45.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 240518168578 2024-09-18T15:32:45.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-201863462917 1-240518168578' 2024-09-18T15:32:45.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:32:45.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:32:45.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509454 2024-09-18T15:32:45.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509454 2024-09-18T15:32:45.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-201863462917 1-240518168578 2-64424509454' 2024-09-18T15:32:45.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:32:45.447 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-201863462917 2024-09-18T15:32:45.447 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:32:45.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:32:45.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-201863462917 2024-09-18T15:32:45.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:32:45.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=201863462917 2024-09-18T15:32:45.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 201863462917' 2024-09-18T15:32:45.453 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 201863462917 2024-09-18T15:32:45.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:32:45.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 201863462917 -lt 201863462917 2024-09-18T15:32:45.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:32:45.808 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-240518168578 2024-09-18T15:32:45.808 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:32:45.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:32:45.811 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-240518168578 2024-09-18T15:32:45.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:32:45.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=240518168578 2024-09-18T15:32:45.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 240518168578' 2024-09-18T15:32:45.814 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 240518168578 2024-09-18T15:32:45.815 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:32:46.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 240518168578 -lt 240518168578 2024-09-18T15:32:46.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:32:46.165 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509454 2024-09-18T15:32:46.165 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:32:46.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:32:46.169 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509454 2024-09-18T15:32:46.169 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:32:46.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509454 2024-09-18T15:32:46.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-09-18T15:32:46.172 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509454 2024-09-18T15:32:46.172 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:32:46.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509454 -lt 64424509454 2024-09-18T15:32:46.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:32:46.520 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:46.520 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:32:46.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:32:46.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:32:46.905 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:32:46.905 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:32:46.905 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:32:46.905 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:32:46.906 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:32:46.906 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:32:47.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:32:47.252 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:32:47.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:47.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:32:47.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:32:47.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:32:47.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:32:47.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-18T15:32:47.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objname=EOBJ4 2024-09-18T15:32:47.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-09-18T15:32:47.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:32:47.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2024-09-18T15:32:47.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ4 2024-09-18T15:32:47.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:32:47.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:32:47.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:32:48.158 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:32:48.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:32:48.644 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:32:48.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:32:48.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:32:48.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:32:48.818 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: expr 4 % 2 2024-09-18T15:32:48.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: local osd=0 2024-09-18T15:32:48.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: case $i in 2024-09-18T15:32:48.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-09-18T15:32:48.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3660: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-09-18T15:32:49.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: echo -n bad-val 2024-09-18T15:32:49.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-18T15:32:49.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:32:49.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:32:49.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:32:49.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:32:49.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-18T15:32:49.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:32:49.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:32:49.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:32:49.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:32:49.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:32:49.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:32:49.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:32:49.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:32:49.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:32:49.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:32:49.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:32:49.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-18T15:32:49.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:32:49.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:32:49.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:32:49.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:32:49.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:32:49.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-18T15:32:50.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-18T15:32:50.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:32:50.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:32:50.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:32:50.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:32:50.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:32:50.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:32:50.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:32:50.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:32:50.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:32:50.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:32:50.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:32:50.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:32:50.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:32:50.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:32:50.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:32:50.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:32:50.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:32:50.972 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:32:50.972 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:32:50.973 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:32:50.974 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:32:50.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:32:50.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:32:50.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:32:50.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:32:50.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:32:50.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:32:50.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:32:50.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:32:50.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:32:50.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:32:50.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:32:50.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:32:50.975 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:32:50.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:32:50.976 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:32:50.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:32:50.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:32:50.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:32:50.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:32:51.005 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:50.998+0000 7f456b735540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:32:51.009 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:51.002+0000 7f456b735540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:32:51.016 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:51.002+0000 7f456b735540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:32:51.016 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:51.006+0000 7f456b735540 -1 Falling back to public interface 2024-09-18T15:32:51.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:32:51.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:32:51.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:32:51.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:32:51.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:32:51.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:32:51.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:32:51.291 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:32:51.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:32:51.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:32:51.352 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:51.346+0000 7f456b735540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:32:51.360 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:51.354+0000 7f456b735540 -1 journal do_read_entry(1187840): bad header magic 2024-09-18T15:32:51.360 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:51.354+0000 7f456b735540 -1 journal do_read_entry(1187840): bad header magic 2024-09-18T15:32:51.384 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:51.374+0000 7f456b735540 -1 osd.0 60 log_to_monitors true 2024-09-18T15:32:51.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:32:52.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:32:52.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:32:52.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:32:52.625 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:32:52.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:32:52.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:32:53.078 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 65 up_thru 65 down_at 61 last_clean_interval [47,60) [v2:127.0.0.1:6800/328656,v1:127.0.0.1:6801/328656] [v2:127.0.0.1:6802/328656,v1:127.0.0.1:6803/328656] exists,up 9709534c-708f-4053-bc68-1443befb5261 2024-09-18T15:32:53.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:32:53.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:32:53.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:32:53.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:32:53.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:32:53.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:32:53.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:32:53.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:32:53.080 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:32:53.081 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:32:53.081 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:32:53.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:32:53.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:32:53.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:32:53.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:32:53.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:32:53.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:32:53.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:32:53.306 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:32:53.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:32:53.657 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:32:53.657 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:32:53.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:32:53.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:32:53.657 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:32:53.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=279172874242 2024-09-18T15:32:53.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 279172874242 2024-09-18T15:32:53.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-279172874242' 2024-09-18T15:32:53.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:32:53.829 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:32:53.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=240518168581 2024-09-18T15:32:53.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 240518168581 2024-09-18T15:32:53.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-279172874242 1-240518168581' 2024-09-18T15:32:53.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:32:53.977 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:32:54.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509457 2024-09-18T15:32:54.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509457 2024-09-18T15:32:54.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-279172874242 1-240518168581 2-64424509457' 2024-09-18T15:32:54.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:32:54.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-279172874242 2024-09-18T15:32:54.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:32:54.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:32:54.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-279172874242 2024-09-18T15:32:54.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:32:54.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=279172874242 2024-09-18T15:32:54.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 279172874242' 2024-09-18T15:32:54.124 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 279172874242 2024-09-18T15:32:54.124 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:32:54.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 279172874242 2024-09-18T15:32:54.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:32:55.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:32:55.463 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:32:55.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 279172874242 -lt 279172874242 2024-09-18T15:32:55.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:32:55.786 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-240518168581 2024-09-18T15:32:55.786 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:32:55.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:32:55.790 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-240518168581 2024-09-18T15:32:55.790 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:32:55.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=240518168581 2024-09-18T15:32:55.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 240518168581' 2024-09-18T15:32:55.792 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 240518168581 2024-09-18T15:32:55.793 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:32:56.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 240518168581 -lt 240518168581 2024-09-18T15:32:56.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:32:56.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509457 2024-09-18T15:32:56.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:32:56.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:32:56.100 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509457 2024-09-18T15:32:56.100 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:32:56.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509457 2024-09-18T15:32:56.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-09-18T15:32:56.103 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509457 2024-09-18T15:32:56.103 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:32:56.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509457 -lt 64424509457 2024-09-18T15:32:56.456 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:32:56.457 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:56.457 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:32:56.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:32:56.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:32:56.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:32:56.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:32:56.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:32:56.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:32:56.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:32:56.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:32:57.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:32:57.116 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:32:57.116 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:57.116 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:32:57.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:32:57.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:32:57.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:32:57.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-18T15:32:57.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:32:57.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:32:57.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-18T15:32:57.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:32:57.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-18T15:32:57.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:32:57.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:32:57.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-18T15:32:57.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:32:57.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-18T15:32:57.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:32:57.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:32:57.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:32:57.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:32:57.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:32:57.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:32:57.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-18T15:32:57.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:32:57.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:32:57.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-18T15:32:57.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:32:57.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:32:57.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-18T15:32:59.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-18T15:32:59.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:32:59.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:32:59.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T15:32:59.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:32:59.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:32:59.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:32:59.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:32:59.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:32:59.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:32:59.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T15:32:59.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T15:32:59.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:32:59.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:32:59.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:32:59.334 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:32:59.334 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:32:59.334 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:32:59.336 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:32:59.336 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:32:59.336 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:32:59.336 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:32:59.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:32:59.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:32:59.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:32:59.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:32:59.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:32:59.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:32:59.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:32:59.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:32:59.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:32:59.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:32:59.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-18T15:32:59.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T15:32:59.338 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.2 2024-09-18T15:32:59.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:32:59.339 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-18T15:32:59.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T15:32:59.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:32:59.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:32:59.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:32:59.368 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:59.362+0000 7f8863c5e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:32:59.372 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:59.366+0000 7f8863c5e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:32:59.380 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:59.366+0000 7f8863c5e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:32:59.380 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:59.370+0000 7f8863c5e540 -1 Falling back to public interface 2024-09-18T15:32:59.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T15:32:59.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:32:59.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:32:59.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:32:59.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:32:59.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:32:59.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:32:59.703 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:32:59.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:32:59.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:32:59.871 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:59.862+0000 7f8863c5e540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:32:59.874 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:59.866+0000 7f8863c5e540 -1 journal do_read_entry(1474560): bad header magic 2024-09-18T15:32:59.874 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:59.866+0000 7f8863c5e540 -1 journal do_read_entry(1474560): bad header magic 2024-09-18T15:32:59.885 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:32:59.878+0000 7f8863c5e540 -1 osd.2 67 log_to_monitors true 2024-09-18T15:33:00.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:01.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:01.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:01.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:33:01.011 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:33:01.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:01.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:33:01.319 INFO:tasks.workunit.client.0.smithi046.stderr:osd.2 up in weight 1 up_from 72 up_thru 72 down_at 68 last_clean_interval [15,67) [v2:127.0.0.1:6818/329261,v1:127.0.0.1:6819/329261] [v2:127.0.0.1:6820/329261,v1:127.0.0.1:6821/329261] exists,up 3c30fbf0-6eac-4a83-90f1-b24860579ebc 2024-09-18T15:33:01.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:33:01.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:33:01.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:33:01.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:33:01.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:33:01.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:33:01.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:33:01.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:33:01.320 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:33:01.321 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:33:01.321 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:33:01.322 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:33:01.322 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:33:01.322 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:33:01.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:33:01.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:33:01.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:33:01.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:33:01.547 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:33:01.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:33:01.967 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:33:01.967 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:33:01.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:33:01.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:33:01.968 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:33:02.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=279172874245 2024-09-18T15:33:02.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 279172874245 2024-09-18T15:33:02.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-279172874245' 2024-09-18T15:33:02.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:33:02.107 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:33:02.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=240518168583 2024-09-18T15:33:02.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 240518168583 2024-09-18T15:33:02.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-279172874245 1-240518168583' 2024-09-18T15:33:02.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:33:02.246 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:33:02.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=309237645314 2024-09-18T15:33:02.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 309237645314 2024-09-18T15:33:02.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-279172874245 1-240518168583 2-309237645314' 2024-09-18T15:33:02.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:33:02.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-279172874245 2024-09-18T15:33:02.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:33:02.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:33:02.417 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-279172874245 2024-09-18T15:33:02.417 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:33:02.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=279172874245 2024-09-18T15:33:02.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 279172874245' 2024-09-18T15:33:02.419 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 279172874245 2024-09-18T15:33:02.419 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:33:02.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 279172874243 -lt 279172874245 2024-09-18T15:33:02.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:33:03.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:33:03.732 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:33:04.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 279172874245 -lt 279172874245 2024-09-18T15:33:04.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:33:04.047 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-240518168583 2024-09-18T15:33:04.047 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:33:04.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:33:04.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-240518168583 2024-09-18T15:33:04.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:33:04.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=240518168583 2024-09-18T15:33:04.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 240518168583' 2024-09-18T15:33:04.052 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 240518168583 2024-09-18T15:33:04.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:33:04.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 240518168584 -lt 240518168583 2024-09-18T15:33:04.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:33:04.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-309237645314 2024-09-18T15:33:04.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:33:04.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:33:04.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-309237645314 2024-09-18T15:33:04.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:33:04.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=309237645314 2024-09-18T15:33:04.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 309237645314' 2024-09-18T15:33:04.410 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 309237645314 2024-09-18T15:33:04.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:33:04.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 309237645314 -lt 309237645314 2024-09-18T15:33:04.768 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:33:04.768 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:33:04.769 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:33:05.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:33:05.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:33:05.134 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:33:05.134 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:33:05.134 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:33:05.134 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:33:05.134 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:33:05.134 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:33:05.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:33:05.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:33:05.476 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:33:05.476 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:33:05.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:33:05.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:33:05.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:33:05.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3666: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-09-18T15:33:05.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3667: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-18T15:33:05.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:33:05.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:33:05.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-18T15:33:05.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:33:05.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-18T15:33:05.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:33:05.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:33:05.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-18T15:33:05.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:33:05.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-18T15:33:05.844 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:33:05.844 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:33:05.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:33:05.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:33:05.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:33:06.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:33:06.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-18T15:33:06.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:33:06.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:33:06.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-18T15:33:06.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:33:06.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:33:06.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-18T15:33:07.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-18T15:33:07.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:33:07.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:33:07.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T15:33:07.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:33:07.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:33:07.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:33:07.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:33:07.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:33:07.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:33:07.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T15:33:07.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T15:33:07.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:33:07.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:33:07.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:33:07.733 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:33:07.733 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:33:07.733 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:33:07.733 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:33:07.733 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:33:07.733 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:33:07.734 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:33:07.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:33:07.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:33:07.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:33:07.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:33:07.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:33:07.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:33:07.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:33:07.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:33:07.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:33:07.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:33:07.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-18T15:33:07.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T15:33:07.737 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.2 2024-09-18T15:33:07.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:33:07.739 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-18T15:33:07.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T15:33:07.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:33:07.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:33:07.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:33:07.769 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:07.762+0000 7fb570ebc540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:07.772 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:07.762+0000 7fb570ebc540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:07.773 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:07.766+0000 7fb570ebc540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:07.775 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:07.766+0000 7fb570ebc540 -1 Falling back to public interface 2024-09-18T15:33:08.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T15:33:08.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:33:08.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:33:08.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:33:08.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:33:08.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:08.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:33:08.043 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:33:08.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:08.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:33:08.153 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:08.146+0000 7fb570ebc540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:33:08.160 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:08.150+0000 7fb570ebc540 -1 journal do_read_entry(1667072): bad header magic 2024-09-18T15:33:08.160 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:08.150+0000 7fb570ebc540 -1 journal do_read_entry(1667072): bad header magic 2024-09-18T15:33:08.173 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:08.166+0000 7fb570ebc540 -1 osd.2 73 log_to_monitors true 2024-09-18T15:33:08.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:09.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:09.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:09.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:33:09.349 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:33:09.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:09.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:33:09.858 INFO:tasks.workunit.client.0.smithi046.stderr:osd.2 up in weight 1 up_from 78 up_thru 78 down_at 74 last_clean_interval [72,73) [v2:127.0.0.1:6818/329866,v1:127.0.0.1:6819/329866] [v2:127.0.0.1:6820/329866,v1:127.0.0.1:6821/329866] exists,up 3c30fbf0-6eac-4a83-90f1-b24860579ebc 2024-09-18T15:33:09.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:33:09.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:33:09.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:33:09.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:33:09.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:33:09.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:33:09.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:33:09.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:33:09.859 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:33:09.859 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:33:09.859 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:33:09.860 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:33:09.860 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:33:09.860 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:33:10.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:33:10.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:33:10.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:33:10.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:33:10.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:33:10.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:33:10.408 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:33:10.408 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:33:10.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:33:10.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:33:10.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:33:10.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=279172874247 2024-09-18T15:33:10.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 279172874247 2024-09-18T15:33:10.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-279172874247' 2024-09-18T15:33:10.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:33:10.557 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:33:10.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=240518168586 2024-09-18T15:33:10.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 240518168586 2024-09-18T15:33:10.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-279172874247 1-240518168586' 2024-09-18T15:33:10.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:33:10.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:33:10.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=335007449090 2024-09-18T15:33:10.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 335007449090 2024-09-18T15:33:10.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-279172874247 1-240518168586 2-335007449090' 2024-09-18T15:33:10.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:33:10.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-279172874247 2024-09-18T15:33:10.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:33:10.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:33:10.835 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-279172874247 2024-09-18T15:33:10.835 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:33:10.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=279172874247 2024-09-18T15:33:10.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 279172874247' 2024-09-18T15:33:10.837 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 279172874247 2024-09-18T15:33:10.837 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:33:11.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 279172874246 -lt 279172874247 2024-09-18T15:33:11.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:33:12.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:33:12.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:33:12.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 279172874247 -lt 279172874247 2024-09-18T15:33:12.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:33:12.526 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-240518168586 2024-09-18T15:33:12.526 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:33:12.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:33:12.530 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-240518168586 2024-09-18T15:33:12.530 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:33:12.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=240518168586 2024-09-18T15:33:12.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 240518168586' 2024-09-18T15:33:12.532 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 240518168586 2024-09-18T15:33:12.532 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:33:12.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 240518168586 -lt 240518168586 2024-09-18T15:33:12.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:33:12.837 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-335007449090 2024-09-18T15:33:12.837 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:33:12.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:33:12.840 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-335007449090 2024-09-18T15:33:12.840 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:33:12.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=335007449090 2024-09-18T15:33:12.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 335007449090' 2024-09-18T15:33:12.843 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 335007449090 2024-09-18T15:33:12.843 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:33:13.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 335007449090 -lt 335007449090 2024-09-18T15:33:13.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:33:13.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:33:13.183 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:33:13.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:33:13.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:33:13.640 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:33:13.640 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:33:13.640 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:33:13.640 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:33:13.640 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:33:13.640 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:33:13.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:33:13.954 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:33:13.954 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:33:13.954 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:33:14.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:33:14.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:33:14.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:33:14.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-09-18T15:33:14.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-18T15:33:14.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objname=EOBJ5 2024-09-18T15:33:14.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-09-18T15:33:14.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:33:14.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2024-09-18T15:33:14.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ5 2024-09-18T15:33:14.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:33:14.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:33:14.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:33:14.986 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:33:14.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:33:15.506 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:33:15.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:33:15.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:33:15.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:33:15.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: expr 5 % 2 2024-09-18T15:33:15.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: local osd=1 2024-09-18T15:33:15.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: case $i in 2024-09-18T15:33:15.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3673: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-09-18T15:33:15.726 INFO:tasks.workunit.client.0.smithi046.stderr:2+0 records in 2024-09-18T15:33:15.726 INFO:tasks.workunit.client.0.smithi046.stderr:2+0 records out 2024-09-18T15:33:15.726 INFO:tasks.workunit.client.0.smithi046.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000265688 s, 15.4 MB/s 2024-09-18T15:33:15.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3674: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:33:15.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:33:15.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:33:15.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:33:15.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:33:15.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:33:15.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:33:15.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:33:15.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:33:15.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:33:15.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:33:15.727 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:33:15.728 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:33:15.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:33:15.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:33:15.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:33:16.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:33:16.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:33:16.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:33:16.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:33:16.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:33:16.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:33:16.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:33:16.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:33:17.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:33:17.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:33:17.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:33:17.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:33:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:33:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:33:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:33:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:33:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:33:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:33:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:33:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:33:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:33:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:33:17.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:33:17.257 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:33:17.257 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:33:17.257 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:33:17.258 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:33:17.258 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:33:17.258 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:33:17.259 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:33:17.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:33:17.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:33:17.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:33:17.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:33:17.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:33:17.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:33:17.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:33:17.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:33:17.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:33:17.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:33:17.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:33:17.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:33:17.261 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:33:17.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:33:17.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:33:17.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:33:17.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:33:17.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:33:17.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:33:17.292 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:17.286+0000 7f7256343540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:17.296 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:17.286+0000 7f7256343540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:17.297 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:17.290+0000 7f7256343540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:17.301 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:17.294+0000 7f7256343540 -1 Falling back to public interface 2024-09-18T15:33:17.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:33:17.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:33:17.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:33:17.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:33:17.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:33:17.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:17.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:33:17.568 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:33:17.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:17.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:33:17.810 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:17.802+0000 7f7256343540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:33:17.819 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:17.810+0000 7f7256343540 -1 journal do_read_entry(2011136): bad header magic 2024-09-18T15:33:17.819 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:17.810+0000 7f7256343540 -1 journal do_read_entry(2011136): bad header magic 2024-09-18T15:33:17.830 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:17.822+0000 7f7256343540 -1 osd.1 82 log_to_monitors true 2024-09-18T15:33:17.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:18.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:18.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:18.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:33:18.929 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:33:18.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:18.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:33:19.377 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 86 up_thru 86 down_at 83 last_clean_interval [56,82) [v2:127.0.0.1:6810/330540,v1:127.0.0.1:6811/330540] [v2:127.0.0.1:6812/330540,v1:127.0.0.1:6813/330540] exists,up a98783e8-5660-4fe2-8f10-19ac7db7df61 2024-09-18T15:33:19.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:33:19.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:33:19.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:33:19.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:33:19.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:33:19.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:33:19.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:33:19.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:33:19.378 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:33:19.379 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:33:19.379 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:33:19.380 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:33:19.380 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:33:19.380 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:33:19.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:33:19.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:33:19.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:33:19.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:33:19.613 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:33:19.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:33:19.929 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:33:19.929 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:33:19.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:33:19.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:33:19.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:33:20.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=279172874250 2024-09-18T15:33:20.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 279172874250 2024-09-18T15:33:20.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-279172874250' 2024-09-18T15:33:20.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:33:20.115 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:33:20.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=369367187458 2024-09-18T15:33:20.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 369367187458 2024-09-18T15:33:20.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-279172874250 1-369367187458' 2024-09-18T15:33:20.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:33:20.200 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:33:20.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=335007449093 2024-09-18T15:33:20.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 335007449093 2024-09-18T15:33:20.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-279172874250 1-369367187458 2-335007449093' 2024-09-18T15:33:20.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:33:20.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-279172874250 2024-09-18T15:33:20.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:33:20.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:33:20.348 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-279172874250 2024-09-18T15:33:20.349 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:33:20.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=279172874250 2024-09-18T15:33:20.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 279172874250' 2024-09-18T15:33:20.351 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 279172874250 2024-09-18T15:33:20.352 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:33:20.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 279172874249 -lt 279172874250 2024-09-18T15:33:20.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:33:21.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:33:21.654 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:33:22.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 279172874250 -lt 279172874250 2024-09-18T15:33:22.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:33:22.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-369367187458 2024-09-18T15:33:22.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:33:22.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:33:22.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-369367187458 2024-09-18T15:33:22.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:33:22.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=369367187458 2024-09-18T15:33:22.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 369367187458' 2024-09-18T15:33:22.020 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 369367187458 2024-09-18T15:33:22.021 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:33:22.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 369367187458 -lt 369367187458 2024-09-18T15:33:22.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:33:22.389 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-335007449093 2024-09-18T15:33:22.389 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:33:22.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:33:22.393 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-335007449093 2024-09-18T15:33:22.393 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:33:22.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=335007449093 2024-09-18T15:33:22.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 335007449093' 2024-09-18T15:33:22.396 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 335007449093 2024-09-18T15:33:22.397 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:33:22.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 335007449093 -lt 335007449093 2024-09-18T15:33:22.735 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:33:22.736 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:33:22.736 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:33:23.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:33:23.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:33:23.168 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:33:23.168 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:33:23.168 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:33:23.168 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:33:23.168 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:33:23.168 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:33:23.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:33:23.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:33:23.510 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:33:23.510 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:33:23.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:33:23.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:33:23.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:33:23.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-18T15:33:23.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objname=EOBJ6 2024-09-18T15:33:23.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-09-18T15:33:23.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:33:23.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2024-09-18T15:33:23.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ6 2024-09-18T15:33:23.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:33:23.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:33:23.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:33:24.322 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:33:24.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:33:24.825 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:33:24.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:33:24.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:33:24.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:33:25.040 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: expr 6 % 2 2024-09-18T15:33:25.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: local osd=0 2024-09-18T15:33:25.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: case $i in 2024-09-18T15:33:25.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-18T15:33:25.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:33:25.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:33:25.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:33:25.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:33:25.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-18T15:33:25.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:33:25.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:33:25.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:33:25.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:33:25.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:33:25.044 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:33:25.044 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:33:25.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:33:25.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:33:25.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:33:25.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:33:25.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-18T15:33:25.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:33:25.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:33:25.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:33:25.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:33:25.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:33:25.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-09-18T15:33:26.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-18T15:33:26.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:33:26.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:33:26.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:33:26.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:33:26.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:33:26.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:33:26.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:33:26.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:33:26.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:33:26.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:33:26.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:33:26.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:33:26.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:33:26.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:33:26.609 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:33:26.609 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:33:26.609 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:33:26.610 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:33:26.610 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:33:26.610 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:33:26.610 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:33:26.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:33:26.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:33:26.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:33:26.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:33:26.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:33:26.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:33:26.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:33:26.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:33:26.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:33:26.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:33:26.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:33:26.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:33:26.612 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:33:26.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:33:26.613 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:33:26.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:33:26.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:33:26.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:33:26.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:33:26.643 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:26.634+0000 7f29f5bec540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:26.646 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:26.638+0000 7f29f5bec540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:26.648 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:26.642+0000 7f29f5bec540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:26.650 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:26.642+0000 7f29f5bec540 -1 Falling back to public interface 2024-09-18T15:33:26.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:33:26.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:33:26.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:33:26.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:33:26.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:33:26.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:26.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:33:26.935 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:33:26.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:26.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:33:26.963 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:26.954+0000 7f29f5bec540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:33:26.963 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:26.954+0000 7f29f5bec540 -1 journal do_read_entry(2174976): bad header magic 2024-09-18T15:33:26.963 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:26.954+0000 7f29f5bec540 -1 journal do_read_entry(2174976): bad header magic 2024-09-18T15:33:26.974 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:26.966+0000 7f29f5bec540 -1 osd.0 90 log_to_monitors true 2024-09-18T15:33:27.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:28.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:28.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:28.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:33:28.239 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:33:28.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:28.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:33:28.646 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 94 up_thru 94 down_at 91 last_clean_interval [65,90) [v2:127.0.0.1:6800/331212,v1:127.0.0.1:6801/331212] [v2:127.0.0.1:6802/331212,v1:127.0.0.1:6803/331212] exists,up 9709534c-708f-4053-bc68-1443befb5261 2024-09-18T15:33:28.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:33:28.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:33:28.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:33:28.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:33:28.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:33:28.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:33:28.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:33:28.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:33:28.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:33:28.648 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:33:28.648 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:33:28.649 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:33:28.649 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:33:28.649 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:33:28.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:33:28.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:33:28.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:33:28.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:33:28.880 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:33:29.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:33:29.186 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:33:29.186 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:33:29.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:33:29.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:33:29.186 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:33:29.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=403726925826 2024-09-18T15:33:29.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 403726925826 2024-09-18T15:33:29.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-403726925826' 2024-09-18T15:33:29.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:33:29.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:33:29.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=369367187461 2024-09-18T15:33:29.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 369367187461 2024-09-18T15:33:29.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-403726925826 1-369367187461' 2024-09-18T15:33:29.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:33:29.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:33:29.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=335007449096 2024-09-18T15:33:29.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 335007449096 2024-09-18T15:33:29.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-403726925826 1-369367187461 2-335007449096' 2024-09-18T15:33:29.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:33:29.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-403726925826 2024-09-18T15:33:29.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:33:29.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:33:29.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-403726925826 2024-09-18T15:33:29.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:33:29.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=403726925826 2024-09-18T15:33:29.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 403726925826' 2024-09-18T15:33:29.715 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 403726925826 2024-09-18T15:33:29.716 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:33:30.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 403726925826 -lt 403726925826 2024-09-18T15:33:30.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:33:30.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-369367187461 2024-09-18T15:33:30.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:33:30.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:33:30.034 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-369367187461 2024-09-18T15:33:30.034 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:33:30.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=369367187461 2024-09-18T15:33:30.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 369367187461' 2024-09-18T15:33:30.037 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 369367187461 2024-09-18T15:33:30.038 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:33:30.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 369367187461 -lt 369367187461 2024-09-18T15:33:30.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:33:30.395 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-335007449096 2024-09-18T15:33:30.395 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:33:30.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:33:30.399 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-335007449096 2024-09-18T15:33:30.399 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:33:30.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=335007449096 2024-09-18T15:33:30.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 335007449096' 2024-09-18T15:33:30.401 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 335007449096 2024-09-18T15:33:30.402 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:33:30.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 335007449095 -lt 335007449096 2024-09-18T15:33:30.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:33:31.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:33:31.742 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:33:32.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 335007449096 -lt 335007449096 2024-09-18T15:33:32.067 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:33:32.068 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:33:32.068 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:33:32.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:33:32.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:33:32.438 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:33:32.438 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:33:32.438 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:33:32.438 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:33:32.439 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:33:32.439 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:33:32.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:33:32.746 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:33:32.746 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:33:32.746 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:33:33.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:33:33.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:33:33.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:33:33.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3679: corrupt_scrub_erasure: echo -n bad-val 2024-09-18T15:33:33.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-18T15:33:33.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:33:33.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:33:33.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:33:33.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:33:33.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-18T15:33:33.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:33:33.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:33:33.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:33:33.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:33:33.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:33:33.113 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:33:33.113 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:33:33.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:33:33.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:33:33.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:33:33.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:33:33.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-18T15:33:33.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:33:33.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:33:33.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:33:33.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:33:33.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:33:33.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-18T15:33:34.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:33:34.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:33:34.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:33:34.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:33:34.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:33:34.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:33:34.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:33:34.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:33:34.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:33:34.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:33:34.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:33:34.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:33:34.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:33:34.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:33:34.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:33:34.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:33:34.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:33:34.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:33:34.997 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:33:34.997 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:33:34.997 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:33:34.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:33:34.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:33:34.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:33:34.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:33:34.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:33:34.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:33:34.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:33:34.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:33:34.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:33:34.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:33:34.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:33:34.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:33:34.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:33:35.000 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:33:35.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:33:35.000 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:33:35.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:33:35.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:33:35.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:33:35.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:33:35.031 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:35.022+0000 7f11c5a0c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:35.034 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:35.026+0000 7f11c5a0c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:35.035 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:35.026+0000 7f11c5a0c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:35.038 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:35.030+0000 7f11c5a0c540 -1 Falling back to public interface 2024-09-18T15:33:35.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:33:35.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:33:35.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:33:35.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:33:35.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:33:35.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:35.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:33:35.317 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:33:35.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:35.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:33:35.425 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:35.418+0000 7f11c5a0c540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:33:35.431 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:35.422+0000 7f11c5a0c540 -1 journal do_read_entry(2506752): bad header magic 2024-09-18T15:33:35.431 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:35.422+0000 7f11c5a0c540 -1 journal do_read_entry(2506752): bad header magic 2024-09-18T15:33:35.442 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:35.434+0000 7f11c5a0c540 -1 osd.1 95 log_to_monitors true 2024-09-18T15:33:35.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:36.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:36.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:36.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:33:36.679 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:33:36.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:36.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:33:37.111 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 100 up_thru 100 down_at 96 last_clean_interval [86,95) [v2:127.0.0.1:6810/331817,v1:127.0.0.1:6811/331817] [v2:127.0.0.1:6812/331817,v1:127.0.0.1:6813/331817] exists,up a98783e8-5660-4fe2-8f10-19ac7db7df61 2024-09-18T15:33:37.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:33:37.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:33:37.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:33:37.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:33:37.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:33:37.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:33:37.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:33:37.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:33:37.112 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:33:37.113 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:33:37.113 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:33:37.114 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:33:37.114 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:33:37.114 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:33:37.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:33:37.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:33:37.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:33:37.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:33:37.335 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:33:37.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:33:37.763 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:33:37.763 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:33:37.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:33:37.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:33:37.763 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:33:37.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=403726925829 2024-09-18T15:33:37.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 403726925829 2024-09-18T15:33:37.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-403726925829' 2024-09-18T15:33:37.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:33:37.945 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:33:38.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=429496729602 2024-09-18T15:33:38.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 429496729602 2024-09-18T15:33:38.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-403726925829 1-429496729602' 2024-09-18T15:33:38.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:33:38.083 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:33:38.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=335007449098 2024-09-18T15:33:38.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 335007449098 2024-09-18T15:33:38.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-403726925829 1-429496729602 2-335007449098' 2024-09-18T15:33:38.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:33:38.226 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-403726925829 2024-09-18T15:33:38.226 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:33:38.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:33:38.228 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-403726925829 2024-09-18T15:33:38.229 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:33:38.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=403726925829 2024-09-18T15:33:38.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 403726925829' 2024-09-18T15:33:38.231 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 403726925829 2024-09-18T15:33:38.233 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:33:38.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 403726925828 -lt 403726925829 2024-09-18T15:33:38.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:33:39.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:33:39.600 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:33:39.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 403726925829 -lt 403726925829 2024-09-18T15:33:39.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:33:39.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-429496729602 2024-09-18T15:33:39.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:33:39.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:33:39.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-429496729602 2024-09-18T15:33:39.924 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:33:39.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=429496729602 2024-09-18T15:33:39.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 429496729602' 2024-09-18T15:33:39.926 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 429496729602 2024-09-18T15:33:39.927 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:33:40.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 429496729602 -lt 429496729602 2024-09-18T15:33:40.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:33:40.269 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-335007449098 2024-09-18T15:33:40.269 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:33:40.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:33:40.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-335007449098 2024-09-18T15:33:40.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:33:40.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=335007449098 2024-09-18T15:33:40.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 335007449098' 2024-09-18T15:33:40.276 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 335007449098 2024-09-18T15:33:40.276 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:33:40.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 335007449099 -lt 335007449098 2024-09-18T15:33:40.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:33:40.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:33:40.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:33:40.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:33:40.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:33:40.989 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:33:40.989 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:33:40.989 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:33:40.989 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:33:40.989 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:33:40.990 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:33:41.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:33:41.349 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:33:41.350 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:33:41.350 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:33:41.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:33:41.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:33:41.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:33:41.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-18T15:33:41.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objname=EOBJ7 2024-09-18T15:33:41.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-09-18T15:33:41.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:33:41.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2024-09-18T15:33:41.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ7 2024-09-18T15:33:41.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:33:41.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:33:41.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:33:42.182 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:33:42.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:33:42.795 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:33:42.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:33:42.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:33:42.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:33:42.976 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: expr 7 % 2 2024-09-18T15:33:42.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: local osd=1 2024-09-18T15:33:42.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: case $i in 2024-09-18T15:33:42.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-09-18T15:33:42.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3685: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-09-18T15:33:42.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-09-18T15:33:43.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3689: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-18T15:33:43.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:33:43.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:33:43.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:33:43.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:33:43.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-18T15:33:43.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:33:43.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:33:43.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:33:43.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:33:43.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:33:43.178 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:33:43.178 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:33:43.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:33:43.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:33:43.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:33:43.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:33:43.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-18T15:33:43.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:33:43.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:33:43.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:33:43.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:33:43.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:33:43.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-09-18T15:33:44.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-18T15:33:44.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:33:44.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:33:44.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:33:44.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:33:44.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:33:44.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:33:44.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:33:44.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:33:44.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:33:44.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:33:44.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:33:44.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:33:44.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:33:44.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:33:44.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:33:44.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:33:44.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:33:44.884 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:33:44.884 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:33:44.884 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:33:44.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:33:44.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:33:44.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:33:44.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:33:44.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:33:44.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:33:44.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:33:44.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:33:44.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:33:44.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:33:44.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:33:44.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:33:44.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:33:44.887 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:33:44.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:33:44.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:33:44.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:33:44.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:33:44.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:33:44.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:33:44.918 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:44.909+0000 7fc260a24540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:44.922 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:44.913+0000 7fc260a24540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:44.924 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:44.913+0000 7fc260a24540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:44.926 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:44.917+0000 7fc260a24540 -1 Falling back to public interface 2024-09-18T15:33:45.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:33:45.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:33:45.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:33:45.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:33:45.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:33:45.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:45.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:33:45.197 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:33:45.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:45.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:33:45.237 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:45.229+0000 7fc260a24540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:33:45.256 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:45.249+0000 7fc260a24540 -1 journal do_read_entry(2650112): bad header magic 2024-09-18T15:33:45.257 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:45.249+0000 7fc260a24540 -1 journal do_read_entry(2650112): bad header magic 2024-09-18T15:33:45.266 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:45.257+0000 7fc260a24540 -1 osd.0 104 log_to_monitors true 2024-09-18T15:33:45.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:46.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:46.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:46.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:33:46.509 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:33:46.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:46.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:33:46.887 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 108 up_thru 108 down_at 105 last_clean_interval [94,104) [v2:127.0.0.1:6800/332509,v1:127.0.0.1:6801/332509] [v2:127.0.0.1:6802/332509,v1:127.0.0.1:6803/332509] exists,up 9709534c-708f-4053-bc68-1443befb5261 2024-09-18T15:33:46.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:33:46.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:33:46.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:33:46.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:33:46.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:33:46.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:33:46.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:33:46.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:33:46.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:33:46.888 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:33:46.889 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:33:46.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:33:46.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:33:46.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:33:47.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:33:47.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:33:47.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:33:47.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:33:47.116 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:33:47.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:33:47.431 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:33:47.431 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:33:47.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:33:47.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:33:47.432 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:33:47.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=463856467970 2024-09-18T15:33:47.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 463856467970 2024-09-18T15:33:47.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-463856467970' 2024-09-18T15:33:47.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:33:47.612 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:33:47.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=429496729605 2024-09-18T15:33:47.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 429496729605 2024-09-18T15:33:47.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-463856467970 1-429496729605' 2024-09-18T15:33:47.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:33:47.758 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:33:47.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=335007449101 2024-09-18T15:33:47.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 335007449101 2024-09-18T15:33:47.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-463856467970 1-429496729605 2-335007449101' 2024-09-18T15:33:47.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:33:47.896 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-463856467970 2024-09-18T15:33:47.896 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:33:47.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:33:47.899 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-463856467970 2024-09-18T15:33:47.899 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:33:47.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=463856467970 2024-09-18T15:33:47.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 463856467970' 2024-09-18T15:33:47.902 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 463856467970 2024-09-18T15:33:47.902 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:33:48.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 463856467970 2024-09-18T15:33:48.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:33:49.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:33:49.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:33:49.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 463856467970 2024-09-18T15:33:49.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:33:50.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-18T15:33:50.525 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:33:50.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 463856467970 -lt 463856467970 2024-09-18T15:33:50.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:33:50.839 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-429496729605 2024-09-18T15:33:50.839 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:33:50.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:33:50.843 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-429496729605 2024-09-18T15:33:50.843 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:33:50.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=429496729605 2024-09-18T15:33:50.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 429496729605' 2024-09-18T15:33:50.845 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 429496729605 2024-09-18T15:33:50.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:33:51.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 429496729605 -lt 429496729605 2024-09-18T15:33:51.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:33:51.201 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-335007449101 2024-09-18T15:33:51.201 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:33:51.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:33:51.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-335007449101 2024-09-18T15:33:51.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:33:51.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=335007449101 2024-09-18T15:33:51.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 335007449101' 2024-09-18T15:33:51.208 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 335007449101 2024-09-18T15:33:51.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:33:51.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 335007449102 -lt 335007449101 2024-09-18T15:33:51.563 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:33:51.564 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:33:51.564 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:33:51.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:33:51.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:33:51.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:33:51.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:33:51.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:33:51.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:33:51.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:33:51.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:33:52.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:33:52.222 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:33:52.223 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:33:52.223 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:33:52.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:33:52.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:33:52.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:33:52.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-18T15:33:52.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:33:52.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:33:52.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:33:52.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:33:52.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-18T15:33:52.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:33:52.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:33:52.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:33:52.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:33:52.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:33:52.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:33:52.583 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:33:52.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:33:52.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:33:52.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:33:52.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:33:52.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-18T15:33:52.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:33:52.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:33:52.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:33:52.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:33:52.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:33:52.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-18T15:33:54.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-18T15:33:54.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:33:54.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:33:54.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:33:54.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:33:54.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:33:54.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:33:54.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:33:54.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:33:54.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:33:54.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:33:54.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:33:54.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:33:54.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:33:54.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:33:54.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:33:54.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:33:54.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:33:54.365 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:33:54.365 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:33:54.365 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:33:54.365 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:33:54.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:33:54.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:33:54.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:33:54.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:33:54.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:33:54.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:33:54.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:33:54.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:33:54.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:33:54.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:33:54.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:33:54.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:33:54.367 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:33:54.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:33:54.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:33:54.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:33:54.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:33:54.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:33:54.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:33:54.399 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:54.389+0000 7fcf2d003540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:54.404 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:54.393+0000 7fcf2d003540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:54.405 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:54.397+0000 7fcf2d003540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:54.407 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:54.397+0000 7fcf2d003540 -1 Falling back to public interface 2024-09-18T15:33:54.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:33:54.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:33:54.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:33:54.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:33:54.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:33:54.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:54.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:33:54.774 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:33:54.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:54.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:33:54.819 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:54.809+0000 7fcf2d003540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:33:54.831 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:54.821+0000 7fcf2d003540 -1 journal do_read_entry(2867200): bad header magic 2024-09-18T15:33:54.831 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:54.821+0000 7fcf2d003540 -1 journal do_read_entry(2867200): bad header magic 2024-09-18T15:33:54.865 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:33:54.857+0000 7fcf2d003540 -1 osd.0 110 log_to_monitors true 2024-09-18T15:33:55.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:56.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:56.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:56.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:33:56.109 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:33:56.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:56.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:33:56.633 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 115 up_thru 115 down_at 111 last_clean_interval [108,110) [v2:127.0.0.1:6800/333138,v1:127.0.0.1:6801/333138] [v2:127.0.0.1:6802/333138,v1:127.0.0.1:6803/333138] exists,up 9709534c-708f-4053-bc68-1443befb5261 2024-09-18T15:33:56.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:33:56.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:33:56.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:33:56.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:33:56.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:33:56.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:33:56.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:33:56.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:33:56.634 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:33:56.634 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:33:56.635 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:33:56.635 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:33:56.635 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:33:56.635 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:33:56.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:33:56.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:33:56.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:33:56.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:33:56.860 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:33:57.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:33:57.172 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:33:57.172 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:33:57.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:33:57.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:33:57.172 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:33:57.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=493921239042 2024-09-18T15:33:57.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 493921239042 2024-09-18T15:33:57.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-493921239042' 2024-09-18T15:33:57.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:33:57.356 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:33:57.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=429496729608 2024-09-18T15:33:57.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 429496729608 2024-09-18T15:33:57.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-493921239042 1-429496729608' 2024-09-18T15:33:57.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:33:57.496 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:33:57.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=335007449104 2024-09-18T15:33:57.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 335007449104 2024-09-18T15:33:57.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-493921239042 1-429496729608 2-335007449104' 2024-09-18T15:33:57.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:33:57.632 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-493921239042 2024-09-18T15:33:57.633 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:33:57.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:33:57.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-493921239042 2024-09-18T15:33:57.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:33:57.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=493921239042 2024-09-18T15:33:57.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 493921239042' 2024-09-18T15:33:57.639 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 493921239042 2024-09-18T15:33:57.639 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:33:57.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 493921239042 -lt 493921239042 2024-09-18T15:33:57.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:33:57.944 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-429496729608 2024-09-18T15:33:57.944 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:33:57.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:33:57.947 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-429496729608 2024-09-18T15:33:57.948 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:33:57.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=429496729608 2024-09-18T15:33:57.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 429496729608' 2024-09-18T15:33:57.950 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 429496729608 2024-09-18T15:33:57.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:33:58.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 429496729608 -lt 429496729608 2024-09-18T15:33:58.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:33:58.288 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:33:58.288 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-335007449104 2024-09-18T15:33:58.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:33:58.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-335007449104 2024-09-18T15:33:58.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:33:58.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=335007449104 2024-09-18T15:33:58.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 335007449104' 2024-09-18T15:33:58.293 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 335007449104 2024-09-18T15:33:58.293 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:33:58.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 335007449103 -lt 335007449104 2024-09-18T15:33:58.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:33:59.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:33:59.602 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:33:59.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 335007449105 -lt 335007449104 2024-09-18T15:33:59.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:33:59.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:33:59.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:34:00.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:34:00.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:34:00.396 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:34:00.396 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:34:00.396 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:34:00.396 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:34:00.396 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:34:00.397 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:34:00.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:34:00.800 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:34:00.800 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:34:00.801 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:34:01.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:34:01.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:34:01.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:34:01.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3691: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-09-18T15:34:01.169 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3697: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-09-18T15:34:01.170 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-09-18T15:34:01.170 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=EOBJ0 2024-09-18T15:34:01.170 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool EOBJ0 2024-09-18T15:34:01.170 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:34:01.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3697: corrupt_scrub_erasure: local pg=3.0 2024-09-18T15:34:01.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3699: corrupt_scrub_erasure: pg_scrub 3.0 2024-09-18T15:34:01.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: pg_scrub: local pgid=3.0 2024-09-18T15:34:01.518 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: pg_scrub: get_last_scrub_stamp 3.0 2024-09-18T15:34:01.518 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:34:01.518 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:34:01.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:34:01.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:34:01.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: pg_scrub: local last_scrub=2024-09-18T15:32:10.257332+0000 2024-09-18T15:34:01.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_scrub: ceph pg scrub 3.0 2024-09-18T15:34:02.218 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-09-18T15:34:02.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_scrub: wait_for_scrub 3.0 2024-09-18T15:32:10.257332+0000 2024-09-18T15:34:02.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=3.0 2024-09-18T15:34:02.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:32:10.257332+0000 2024-09-18T15:34:02.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:34:02.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:34:02.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:34:02.227 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:34:02.227 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:34:02.228 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:34:02.228 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:34:02.228 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:34:02.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:32:10.257332+0000 '>' 2024-09-18T15:32:10.257332+0000 2024-09-18T15:34:02.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:34:03.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:34:03.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:34:03.528 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:34:03.528 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:34:03.529 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:34:03.529 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:34:03.529 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:34:03.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:32:10.257332+0000 '>' 2024-09-18T15:32:10.257332+0000 2024-09-18T15:34:03.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:34:04.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:34:04.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:34:04.872 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:34:04.873 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:34:04.873 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:34:04.873 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:34:04.873 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:34:05.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:32:10.257332+0000 '>' 2024-09-18T15:32:10.257332+0000 2024-09-18T15:34:05.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:34:06.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:34:06.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:34:06.187 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:34:06.187 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:34:06.187 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:34:06.187 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:34:06.187 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:34:06.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:32:10.257332+0000 '>' 2024-09-18T15:32:10.257332+0000 2024-09-18T15:34:06.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:34:07.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:34:07.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:34:07.526 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:34:07.526 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:34:07.526 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:34:07.527 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:34:07.527 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:34:07.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:32:10.257332+0000 '>' 2024-09-18T15:32:10.257332+0000 2024-09-18T15:34:07.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:34:08.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:34:08.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:34:08.845 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:34:08.845 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:34:08.845 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:34:08.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:34:08.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:34:09.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:32:10.257332+0000 '>' 2024-09-18T15:32:10.257332+0000 2024-09-18T15:34:09.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:34:10.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:34:10.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:34:10.159 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:34:10.159 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:34:10.159 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:34:10.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:34:10.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:34:10.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:32:10.257332+0000 '>' 2024-09-18T15:32:10.257332+0000 2024-09-18T15:34:10.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:34:11.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:34:11.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:34:11.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:34:11.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:34:11.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:34:11.476 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:34:11.476 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:34:11.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:34:09.525691+0000 '>' 2024-09-18T15:32:10.257332+0000 2024-09-18T15:34:11.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:34:11.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3701: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-09-18T15:34:11.854 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3703: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-09-18T15:34:11.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3703: corrupt_scrub_erasure: test 1 = 1 2024-09-18T15:34:11.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3705: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-18T15:34:11.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3705: corrupt_scrub_erasure: test 3.0 = 3.0 2024-09-18T15:34:11.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3707: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-09-18T15:34:11.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3709: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-09-18T15:34:12.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3709: corrupt_scrub_erasure: epoch=115 2024-09-18T15:34:12.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3711: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-18T15:34:12.025 INFO:tasks.workunit.client.0.smithi046.stderr: . as $in 2024-09-18T15:34:12.025 INFO:tasks.workunit.client.0.smithi046.stderr: | if type == "object" then 2024-09-18T15:34:12.025 INFO:tasks.workunit.client.0.smithi046.stderr: reduce keys[] as $key 2024-09-18T15:34:12.025 INFO:tasks.workunit.client.0.smithi046.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-18T15:34:12.025 INFO:tasks.workunit.client.0.smithi046.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-18T15:34:12.025 INFO:tasks.workunit.client.0.smithi046.stderr: else f 2024-09-18T15:34:12.025 INFO:tasks.workunit.client.0.smithi046.stderr: end; 2024-09-18T15:34:12.025 INFO:tasks.workunit.client.0.smithi046.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-18T15:34:12.025 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-18T15:34:12.025 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-18T15:34:12.025 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-18T15:34:12.025 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-18T15:34:12.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3711: corrupt_scrub_erasure: jq .inconsistents 2024-09-18T15:34:12.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3711: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-18T15:34:12.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4331: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-18T15:34:12.079 INFO:tasks.workunit.client.0.smithi046.stderr: . as $in 2024-09-18T15:34:12.080 INFO:tasks.workunit.client.0.smithi046.stderr: | if type == "object" then 2024-09-18T15:34:12.080 INFO:tasks.workunit.client.0.smithi046.stderr: reduce keys[] as $key 2024-09-18T15:34:12.080 INFO:tasks.workunit.client.0.smithi046.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-18T15:34:12.080 INFO:tasks.workunit.client.0.smithi046.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-18T15:34:12.080 INFO:tasks.workunit.client.0.smithi046.stderr: else f 2024-09-18T15:34:12.080 INFO:tasks.workunit.client.0.smithi046.stderr: end; 2024-09-18T15:34:12.080 INFO:tasks.workunit.client.0.smithi046.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-18T15:34:12.080 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-18T15:34:12.080 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-18T15:34:12.080 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-18T15:34:12.080 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-18T15:34:12.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4331: corrupt_scrub_erasure: jq .inconsistents 2024-09-18T15:34:12.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4331: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-18T15:34:12.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-18T15:34:12.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-18T15:34:12.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4333: corrupt_scrub_erasure: test no = yes 2024-09-18T15:34:12.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4338: corrupt_scrub_erasure: test '' = yes 2024-09-18T15:34:12.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4343: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-09-18T15:34:12.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: pg_deep_scrub: local pgid=3.0 2024-09-18T15:34:12.131 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-18T15:34:12.131 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:34:12.131 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:34:12.132 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:34:12.132 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-18T15:34:12.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: pg_deep_scrub: local last_scrub=2024-09-18T15:32:10.257332+0000 2024-09-18T15:34:12.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-09-18T15:34:12.727 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-09-18T15:34:12.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: pg_deep_scrub: wait_for_scrub 3.0 2024-09-18T15:32:10.257332+0000 last_deep_scrub_stamp 2024-09-18T15:34:12.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=3.0 2024-09-18T15:34:12.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:32:10.257332+0000 2024-09-18T15:34:12.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-09-18T15:34:12.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:34:12.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:34:12.737 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-18T15:34:12.737 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:34:12.737 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:34:12.737 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:34:12.737 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-18T15:34:13.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:32:10.257332+0000 '>' 2024-09-18T15:32:10.257332+0000 2024-09-18T15:34:13.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:34:14.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:34:14.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:34:14.034 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-18T15:34:14.034 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:34:14.035 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:34:14.035 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:34:14.035 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-18T15:34:14.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:32:10.257332+0000 '>' 2024-09-18T15:32:10.257332+0000 2024-09-18T15:34:14.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:34:15.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:34:15.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:34:15.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-18T15:34:15.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:34:15.356 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:34:15.356 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:34:15.356 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-18T15:34:15.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:32:10.257332+0000 '>' 2024-09-18T15:32:10.257332+0000 2024-09-18T15:34:15.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:34:16.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:34:16.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:34:16.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-18T15:34:16.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:34:16.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:34:16.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:34:16.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-18T15:34:16.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:32:10.257332+0000 '>' 2024-09-18T15:32:10.257332+0000 2024-09-18T15:34:16.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:34:17.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:34:17.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:34:17.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-18T15:34:17.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:34:17.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:34:17.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:34:17.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-18T15:34:18.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:32:10.257332+0000 '>' 2024-09-18T15:32:10.257332+0000 2024-09-18T15:34:18.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:34:19.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:34:19.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:34:19.305 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-18T15:34:19.305 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:34:19.306 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:34:19.306 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:34:19.306 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-18T15:34:19.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:32:10.257332+0000 '>' 2024-09-18T15:32:10.257332+0000 2024-09-18T15:34:19.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:34:20.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:34:20.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:34:20.620 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-18T15:34:20.620 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:34:20.620 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:34:20.620 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:34:20.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-18T15:34:20.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:32:10.257332+0000 '>' 2024-09-18T15:32:10.257332+0000 2024-09-18T15:34:20.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:34:21.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:34:21.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:34:21.937 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-18T15:34:21.937 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:34:21.937 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:34:21.938 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:34:21.938 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-18T15:34:22.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:34:16.583161+0000 '>' 2024-09-18T15:32:10.257332+0000 2024-09-18T15:34:22.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:34:22.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4345: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-09-18T15:34:22.327 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4347: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-09-18T15:34:22.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4347: corrupt_scrub_erasure: test 1 = 1 2024-09-18T15:34:22.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4349: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-18T15:34:22.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4349: corrupt_scrub_erasure: test 3.0 = 3.0 2024-09-18T15:34:22.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4351: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-09-18T15:34:22.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4353: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-09-18T15:34:22.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4353: corrupt_scrub_erasure: epoch=115 2024-09-18T15:34:22.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4355: corrupt_scrub_erasure: '[' false = true ']' 2024-09-18T15:34:22.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5012: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-18T15:34:22.501 INFO:tasks.workunit.client.0.smithi046.stderr: . as $in 2024-09-18T15:34:22.501 INFO:tasks.workunit.client.0.smithi046.stderr: | if type == "object" then 2024-09-18T15:34:22.501 INFO:tasks.workunit.client.0.smithi046.stderr: reduce keys[] as $key 2024-09-18T15:34:22.501 INFO:tasks.workunit.client.0.smithi046.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-18T15:34:22.501 INFO:tasks.workunit.client.0.smithi046.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-18T15:34:22.501 INFO:tasks.workunit.client.0.smithi046.stderr: else f 2024-09-18T15:34:22.501 INFO:tasks.workunit.client.0.smithi046.stderr: end; 2024-09-18T15:34:22.501 INFO:tasks.workunit.client.0.smithi046.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-18T15:34:22.501 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-18T15:34:22.501 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-18T15:34:22.501 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-18T15:34:22.502 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-18T15:34:22.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5012: corrupt_scrub_erasure: jq .inconsistents 2024-09-18T15:34:22.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5012: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-18T15:34:22.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5745: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-18T15:34:22.563 INFO:tasks.workunit.client.0.smithi046.stderr: . as $in 2024-09-18T15:34:22.563 INFO:tasks.workunit.client.0.smithi046.stderr: | if type == "object" then 2024-09-18T15:34:22.563 INFO:tasks.workunit.client.0.smithi046.stderr: reduce keys[] as $key 2024-09-18T15:34:22.563 INFO:tasks.workunit.client.0.smithi046.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-18T15:34:22.564 INFO:tasks.workunit.client.0.smithi046.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-18T15:34:22.564 INFO:tasks.workunit.client.0.smithi046.stderr: else f 2024-09-18T15:34:22.564 INFO:tasks.workunit.client.0.smithi046.stderr: end; 2024-09-18T15:34:22.564 INFO:tasks.workunit.client.0.smithi046.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-18T15:34:22.564 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-18T15:34:22.564 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-18T15:34:22.564 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-18T15:34:22.564 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-18T15:34:22.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5745: corrupt_scrub_erasure: jq .inconsistents 2024-09-18T15:34:22.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5745: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-18T15:34:22.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5746: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-18T15:34:22.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-18T15:34:22.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5747: corrupt_scrub_erasure: test no = yes 2024-09-18T15:34:22.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5758: corrupt_scrub_erasure: test '' = yes 2024-09-18T15:34:22.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5763: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-09-18T15:34:23.055 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'ecpool' removed 2024-09-18T15:34:23.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T15:34:23.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:34:23.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:34:23.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:34:23.068 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:34:23.068 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:34:23.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:34:23.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:34:23.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:34:23.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:34:23.192 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:34:23.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:34:23.195 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:34:23.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:34:23.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:34:23.198 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:34:23.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:34:23.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:34:23.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:34:23.202 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:34:23.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:34:23.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:34:23.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:34:23.239 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:34:23.240 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:34:23.240 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:34:23.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:34:23.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:34:23.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:34:23.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T15:34:23.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T15:34:23.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T15:34:23.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T15:34:23.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:34:23.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:34:23.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:34:23.243 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:34:23.243 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:34:23.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:34:23.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:34:23.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:34:23.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:34:23.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:34:23.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:34:23.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:34:23.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:34:23.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:34:23.255 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:34:23.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:34:23.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:34:23.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:34:23.258 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:34:23.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:34:23.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:34:23.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:34:23.265 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:34:23.265 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:34:23.265 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:34:23.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:34:23.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:34:23.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:34:23.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T15:34:23.270 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:34:23.270 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:34:23.270 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:34:23.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T15:34:23.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:34:23.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:34:23.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:34:23.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T15:34:23.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_corrupt_scrub_erasure_overwrites td/osd-scrub-repair 2024-09-18T15:34:23.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5771: TEST_corrupt_scrub_erasure_overwrites: '[' true = true ']' 2024-09-18T15:34:23.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5772: TEST_corrupt_scrub_erasure_overwrites: corrupt_scrub_erasure td/osd-scrub-repair true 2024-09-18T15:34:23.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-09-18T15:34:23.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: local allow_overwrites=true 2024-09-18T15:34:23.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3615: corrupt_scrub_erasure: local poolname=ecpool 2024-09-18T15:34:23.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local total_objs=7 2024-09-18T15:34:23.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-09-18T15:34:23.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T15:34:23.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:34:23.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:34:23.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:34:23.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T15:34:23.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-18T15:34:23.659 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:34:23.659 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:34:23.659 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:34:23.660 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:34:23.660 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:34:23.660 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:34:23.660 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:34:23.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T15:34:23.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:34:23.950 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:34:23.950 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:34:23.950 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:34:23.950 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:34:23.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:34:23.951 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:34:23.951 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:34:23.952 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:34:23.952 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:34:23.953 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:34:23.953 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:34:23.953 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:34:23.953 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:34:23.954 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T15:34:24.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:34:24.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:34:24.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:34:24.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:34:24.051 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:34:24.051 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:34:24.051 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:34:24.051 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:34:24.052 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:34:24.052 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:34:24.052 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:34:24.052 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:34:24.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:34:24.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T15:34:24.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3619: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-09-18T15:34:24.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T15:34:24.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:34:24.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:34:24.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:34:24.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T15:34:24.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:34:25.006 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:34:25.006 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:34:25.006 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:34:25.007 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:34:25.007 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:34:25.007 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:34:25.007 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:34:25.009 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:34:25.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:34:25.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3620: corrupt_scrub_erasure: seq 0 2 2024-09-18T15:34:25.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3620: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-18T15:34:25.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: '[' true = true ']' 2024-09-18T15:34:25.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-09-18T15:34:25.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:34:25.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:34:25.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:34:25.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:34:25.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:34:25.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:34:25.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:34:25.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:34:25.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:34:25.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:34:25.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:34:25.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:34:25.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:34:25.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:34:25.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:34:25.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:34:25.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:34:25.055 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:34:25.055 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:34:25.055 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:34:25.055 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:34:25.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:34:25.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:34:25.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:34:25.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:34:25.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:34:25.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:34:25.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:34:25.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:34:25.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:34:25.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:34:25.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:34:25.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:34:25.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:34:25.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:34:25.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d366d8b9-5243-44c9-8553-5749a6e3a9c7 2024-09-18T15:34:25.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d366d8b9-5243-44c9-8553-5749a6e3a9c7' 2024-09-18T15:34:25.058 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 d366d8b9-5243-44c9-8553-5749a6e3a9c7 2024-09-18T15:34:25.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:34:25.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAB8+pmOAZdBBAAsiAmY1n/5+ZDDRyeukaSVA== 2024-09-18T15:34:25.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAB8+pmOAZdBBAAsiAmY1n/5+ZDDRyeukaSVA=="}' 2024-09-18T15:34:25.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d366d8b9-5243-44c9-8553-5749a6e3a9c7 -i td/osd-scrub-repair/0/new.json 2024-09-18T15:34:25.350 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:34:25.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-18T15:34:25.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAB8+pmOAZdBBAAsiAmY1n/5+ZDDRyeukaSVA== --osd-uuid d366d8b9-5243-44c9-8553-5749a6e3a9c7 2024-09-18T15:34:25.398 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:34:25.389+0000 7f5c85c83540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:34:25.443 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:34:25.433+0000 7f5c85c83540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:34:25.445 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:34:25.437+0000 7f5c85c83540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:34:25.446 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:34:25.437+0000 7f5c85c83540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:34:25.446 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:34:25.437+0000 7f5c85c83540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:34:25.446 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:34:25.437+0000 7f5c85c83540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:34:25.490 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:34:25.481+0000 7f5c85c83540 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-18T15:34:31.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T15:34:31.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:34:31.687 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T15:34:31.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:34:31.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:34:32.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:34:32.071 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T15:34:32.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:34:32.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:34:32.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:34:32.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:34:32.103 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:34:32.093+0000 7fd846f3c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:34:32.106 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:34:32.097+0000 7fd846f3c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:34:32.108 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:34:32.097+0000 7fd846f3c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:34:32.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:34:32.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:34:32.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:34:32.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:34:32.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:34:32.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:32.372 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:34:32.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:34:32.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:32.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:34:32.654 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:34:32.645+0000 7fd846f3c540 -1 Falling back to public interface 2024-09-18T15:34:32.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:33.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:33.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:33.682 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:34:33.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:34:33.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:33.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:34:33.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:34.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:34.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:34.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:34:34.999 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:34:34.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:34.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:34:35.296 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:34:35.289+0000 7fd846f3c540 -1 osd.0 0 log_to_monitors true 2024-09-18T15:34:35.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:36.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:36.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:36.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:34:36.322 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:34:36.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:36.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:34:36.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:37.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:37.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:37.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:34:37.638 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:34:37.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:37.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:34:37.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:38.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:38.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:38.989 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:34:38.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:34:38.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:38.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:34:39.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:40.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:40.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:40.311 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:34:40.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:34:40.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:40.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:34:40.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:41.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:41.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:41.626 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:34:41.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:34:41.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:41.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:34:42.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:43.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:43.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:43.039 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:34:43.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:34:43.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:43.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:34:43.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:44.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:44.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:44.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:34:44.353 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:34:44.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:44.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:34:44.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:45.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:45.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:45.669 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:34:45.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:34:45.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:45.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:34:46.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:47.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:47.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:47.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:34:47.011 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:34:47.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:47.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:34:47.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:48.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:48.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:48.501 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:34:48.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:34:48.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:48.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:34:49.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:50.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:50.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:50.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:34:50.035 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:34:50.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:50.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:34:50.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:51.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:51.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:51.817 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:34:51.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:34:51.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:51.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:34:52.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:53.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:53.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:53.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:34:53.357 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:34:53.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:53.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:34:53.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:54.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:54.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:54.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:34:54.995 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:34:54.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:54.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:34:55.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:56.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:56.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:56.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:34:56.373 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:34:56.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:56.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:34:56.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:57.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:57.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:57.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:34:57.743 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:34:57.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:57.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:34:58.048 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/335478,v1:127.0.0.1:6803/335478] [v2:127.0.0.1:6804/335478,v1:127.0.0.1:6805/335478] exists,up d366d8b9-5243-44c9-8553-5749a6e3a9c7 2024-09-18T15:34:58.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:34:58.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:34:58.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:34:58.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3620: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-18T15:34:58.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: '[' true = true ']' 2024-09-18T15:34:58.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-09-18T15:34:58.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:34:58.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:34:58.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:34:58.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:34:58.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:34:58.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:34:58.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:34:58.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:34:58.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:34:58.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:34:58.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:34:58.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:34:58.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:34:58.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:34:58.052 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:34:58.052 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:34:58.052 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:34:58.052 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:34:58.052 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:34:58.052 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:34:58.052 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:34:58.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:34:58.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:34:58.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:34:58.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:34:58.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:34:58.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:34:58.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:34:58.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:34:58.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:34:58.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:34:58.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:34:58.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:34:58.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:34:58.055 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:34:58.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=144c133f-b1d4-4eed-ab4d-dcfcf7bf1077 2024-09-18T15:34:58.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 144c133f-b1d4-4eed-ab4d-dcfcf7bf1077' 2024-09-18T15:34:58.057 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 144c133f-b1d4-4eed-ab4d-dcfcf7bf1077 2024-09-18T15:34:58.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:34:58.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAi8+pmZnWKBBAAVF5jGhZnsR6wwPbSH4WqbA== 2024-09-18T15:34:58.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAi8+pmZnWKBBAAVF5jGhZnsR6wwPbSH4WqbA=="}' 2024-09-18T15:34:58.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 144c133f-b1d4-4eed-ab4d-dcfcf7bf1077 -i td/osd-scrub-repair/1/new.json 2024-09-18T15:34:58.482 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:34:58.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-18T15:34:58.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAi8+pmZnWKBBAAVF5jGhZnsR6wwPbSH4WqbA== --osd-uuid 144c133f-b1d4-4eed-ab4d-dcfcf7bf1077 2024-09-18T15:34:58.533 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:34:58.525+0000 7fe70097d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:34:58.548 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:34:58.537+0000 7fe70097d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:34:58.550 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:34:58.541+0000 7fe70097d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:34:58.551 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:34:58.541+0000 7fe70097d540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:34:58.551 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:34:58.541+0000 7fe70097d540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:34:58.551 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:34:58.541+0000 7fe70097d540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:34:58.626 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:34:58.617+0000 7fe70097d540 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-18T15:35:04.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-18T15:35:04.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:35:04.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:35:04.792 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T15:35:04.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:35:05.220 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T15:35:05.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:35:05.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:35:05.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:35:05.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:35:05.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:35:05.251 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:35:05.241+0000 7f307acb0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:35:05.253 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:35:05.245+0000 7f307acb0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:35:05.254 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:35:05.245+0000 7f307acb0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:35:05.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:35:05.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:35:05.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:35:05.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:35:05.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:35:05.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:05.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:35:05.522 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:35:05.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:05.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:35:05.818 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:35:05.809+0000 7f307acb0540 -1 Falling back to public interface 2024-09-18T15:35:05.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:06.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:06.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:06.859 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:35:06.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:35:06.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:06.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:35:07.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:08.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:08.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:08.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:35:08.175 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:35:08.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:08.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:35:08.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:08.578 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:35:08.557+0000 7f307acb0540 -1 osd.1 0 log_to_monitors true 2024-09-18T15:35:09.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:09.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:09.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:35:09.485 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:35:09.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:09.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:35:09.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:10.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:10.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:10.807 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:35:10.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:35:10.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:10.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:35:11.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:12.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:12.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:12.190 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:35:12.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:35:12.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:12.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:35:12.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:13.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:13.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:13.586 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:35:13.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:35:13.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:13.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:35:13.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:14.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:14.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:14.904 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:35:14.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:35:14.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:14.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:35:15.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:16.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:16.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:16.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:35:16.227 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:35:16.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:16.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:35:16.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:17.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:17.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:17.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:35:17.644 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:35:17.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:17.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:35:17.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:18.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:18.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:18.968 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:35:18.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:35:18.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:18.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:35:19.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:20.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:20.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:20.381 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:35:20.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:35:20.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:20.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:35:20.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:21.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:21.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:21.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:35:21.874 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:35:21.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:21.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:35:22.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:23.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:23.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:23.392 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:35:23.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:35:23.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:23.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:35:24.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:25.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:25.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:25.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:35:25.075 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:35:25.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:25.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:35:25.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:26.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:26.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:26.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:35:26.585 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:35:26.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:26.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:35:27.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:28.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:28.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:28.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:35:28.140 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:35:28.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:35:28.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:28.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:29.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:29.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:29.584 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:35:29.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:35:29.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:29.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:35:30.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:31.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:31.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:31.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:35:31.305 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:35:31.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:31.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:35:31.665 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/337264,v1:127.0.0.1:6811/337264] [v2:127.0.0.1:6812/337264,v1:127.0.0.1:6813/337264] exists,up 144c133f-b1d4-4eed-ab4d-dcfcf7bf1077 2024-09-18T15:35:31.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:35:31.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:35:31.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:35:31.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3620: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-18T15:35:31.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: '[' true = true ']' 2024-09-18T15:35:31.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-09-18T15:35:31.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:35:31.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:35:31.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T15:35:31.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:35:31.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:35:31.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:35:31.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:35:31.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:35:31.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:35:31.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T15:35:31.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T15:35:31.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:35:31.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:35:31.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:35:31.669 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:35:31.669 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:35:31.669 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:35:31.670 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:35:31.670 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:35:31.671 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:35:31.671 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:35:31.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:35:31.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:35:31.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:35:31.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:35:31.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:35:31.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:35:31.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:35:31.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:35:31.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:35:31.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:35:31.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:35:31.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:35:31.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-18T15:35:31.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:35:31.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1b922d60-8949-4a5a-9a47-86dba5eb7687 2024-09-18T15:35:31.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 1b922d60-8949-4a5a-9a47-86dba5eb7687' 2024-09-18T15:35:31.675 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 1b922d60-8949-4a5a-9a47-86dba5eb7687 2024-09-18T15:35:31.675 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:35:31.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBD8+pmDrA5KRAA7bWtGxRYgGi+EJL2FW7ClA== 2024-09-18T15:35:31.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBD8+pmDrA5KRAA7bWtGxRYgGi+EJL2FW7ClA=="}' 2024-09-18T15:35:31.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1b922d60-8949-4a5a-9a47-86dba5eb7687 -i td/osd-scrub-repair/2/new.json 2024-09-18T15:35:32.094 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:35:32.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-18T15:35:32.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBD8+pmDrA5KRAA7bWtGxRYgGi+EJL2FW7ClA== --osd-uuid 1b922d60-8949-4a5a-9a47-86dba5eb7687 2024-09-18T15:35:32.149 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:35:32.140+0000 7f78c4c0f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:35:32.153 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:35:32.144+0000 7f78c4c0f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:35:32.155 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:35:32.144+0000 7f78c4c0f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:35:32.156 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:35:32.148+0000 7f78c4c0f540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T15:35:32.156 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:35:32.148+0000 7f78c4c0f540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T15:35:32.157 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:35:32.148+0000 7f78c4c0f540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T15:35:32.235 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:35:32.228+0000 7f78c4c0f540 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-18T15:35:38.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-18T15:35:38.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:35:38.551 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T15:35:38.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T15:35:38.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:35:38.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T15:35:38.939 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T15:35:38.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:35:38.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:35:38.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:35:38.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:35:38.964 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:35:38.952+0000 7fa98fb81540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:35:38.967 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:35:38.956+0000 7fa98fb81540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:35:38.968 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:35:38.960+0000 7fa98fb81540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:35:39.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T15:35:39.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:35:39.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:35:39.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:35:39.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:35:39.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:39.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:35:39.286 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:35:39.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:39.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:35:39.505 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:35:39.496+0000 7fa98fb81540 -1 Falling back to public interface 2024-09-18T15:35:39.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:40.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:40.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:40.594 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:35:40.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:35:40.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:40.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:35:40.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:41.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:41.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:41.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:35:41.903 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:35:41.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:41.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:35:42.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:42.294 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:35:42.284+0000 7fa98fb81540 -1 osd.2 0 log_to_monitors true 2024-09-18T15:35:43.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:43.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:43.228 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:35:43.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:35:43.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:43.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:35:43.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:44.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:44.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:44.551 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:35:44.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:35:44.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:44.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:35:44.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:45.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:45.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:45.871 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:35:45.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:35:45.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:45.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:35:46.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:47.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:47.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:47.194 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:35:47.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:35:47.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:47.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:35:47.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:48.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:48.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:48.615 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:35:48.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:35:48.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:48.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:35:48.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:49.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:49.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:49.972 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:35:49.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:35:49.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:49.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:35:50.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:51.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:51.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:51.307 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:35:51.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:35:51.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:51.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:35:51.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:52.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:52.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:52.616 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:35:52.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:35:52.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:52.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:35:52.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:53.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:54.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:54.192 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:35:54.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:35:54.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:54.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:35:54.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:55.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:55.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:55.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:35:55.305 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:35:55.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:55.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:35:55.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:56.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:56.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:56.650 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:35:56.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:35:56.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:56.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:35:57.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:58.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:58.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:58.091 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:35:58.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:35:58.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:58.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:35:58.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:59.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:59.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:59.591 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:35:59.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:35:59.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:59.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:35:59.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:00.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:00.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:00.940 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:36:00.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:36:00.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:00.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:36:01.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:02.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:02.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:02.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:36:02.364 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:36:02.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:02.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:36:03.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:04.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:04.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:04.033 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:36:04.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:36:04.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:04.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:36:04.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:05.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:05.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:05.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:36:05.491 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:36:05.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:05.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:36:05.824 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/339054,v1:127.0.0.1:6819/339054] [v2:127.0.0.1:6820/339054,v1:127.0.0.1:6821/339054] exists,up 1b922d60-8949-4a5a-9a47-86dba5eb7687 2024-09-18T15:36:05.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:36:05.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:36:05.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:36:05.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3627: corrupt_scrub_erasure: create_rbd_pool 2024-09-18T15:36:05.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T15:36:06.158 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' does not exist 2024-09-18T15:36:06.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T15:36:06.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T15:36:06.757 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' created 2024-09-18T15:36:06.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:36:07.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T15:36:09.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: create_pool foo 1 2024-09-18T15:36:09.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-09-18T15:36:10.360 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'foo' created 2024-09-18T15:36:10.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:36:11.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3630: corrupt_scrub_erasure: create_ec_pool ecpool true k=2 m=1 stripe_unit=2K --force 2024-09-18T15:36:11.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: create_ec_pool: local pool_name=ecpool 2024-09-18T15:36:11.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: create_ec_pool: shift 2024-09-18T15:36:11.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: create_ec_pool: local allow_overwrites=true 2024-09-18T15:36:11.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: create_ec_pool: shift 2024-09-18T15:36:11.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-09-18T15:36:11.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-18T15:36:11.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-18T15:36:12.704 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'ecpool' created 2024-09-18T15:36:12.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:36:13.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: create_ec_pool: '[' true = true ']' 2024-09-18T15:36:13.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-18T15:36:14.287 INFO:tasks.workunit.client.0.smithi046.stderr:set pool 3 allow_ec_overwrites to true 2024-09-18T15:36:14.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: create_ec_pool: wait_for_clean 2024-09-18T15:36:14.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:36:14.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:36:14.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:36:14.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:36:14.301 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:36:14.302 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:36:14.303 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:36:14.303 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:36:14.303 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:36:14.303 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:36:14.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:36:14.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:36:14.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:36:14.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:36:14.533 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:36:14.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:36:14.918 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:36:14.918 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:36:14.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:36:14.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:36:14.918 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:36:15.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836501 2024-09-18T15:36:15.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836501 2024-09-18T15:36:15.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836501' 2024-09-18T15:36:15.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:36:15.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:36:15.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672982 2024-09-18T15:36:15.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672982 2024-09-18T15:36:15.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836501 1-42949672982' 2024-09-18T15:36:15.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:36:15.303 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:36:15.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509453 2024-09-18T15:36:15.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509453 2024-09-18T15:36:15.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836501 1-42949672982 2-64424509453' 2024-09-18T15:36:15.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:36:15.446 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836501 2024-09-18T15:36:15.447 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:36:15.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:36:15.450 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836501 2024-09-18T15:36:15.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:36:15.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836501 2024-09-18T15:36:15.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-18T15:36:15.453 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836501 2024-09-18T15:36:15.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:36:15.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836500 -lt 21474836501 2024-09-18T15:36:15.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:36:16.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:36:16.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:36:17.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836501 -lt 21474836501 2024-09-18T15:36:17.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:36:17.092 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672982 2024-09-18T15:36:17.093 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:36:17.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:36:17.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672982 2024-09-18T15:36:17.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:36:17.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672982 2024-09-18T15:36:17.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-09-18T15:36:17.099 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672982 2024-09-18T15:36:17.100 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:36:17.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672982 -lt 42949672982 2024-09-18T15:36:17.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:36:17.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509453 2024-09-18T15:36:17.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:36:17.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:36:17.429 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509453 2024-09-18T15:36:17.429 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:36:17.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509453 2024-09-18T15:36:17.431 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509453 2024-09-18T15:36:17.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-18T15:36:17.432 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:36:17.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509454 -lt 64424509453 2024-09-18T15:36:17.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:36:17.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:36:17.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:36:18.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:36:18.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:36:18.190 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:36:18.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:36:18.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:36:18.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:36:18.192 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:36:18.192 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:36:18.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:36:18.536 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:36:18.537 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:36:18.537 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:36:18.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:36:18.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:36:18.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:36:18.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: create_ec_pool: return 0 2024-09-18T15:36:18.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3631: corrupt_scrub_erasure: wait_for_clean 2024-09-18T15:36:18.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:36:18.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:36:18.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:36:18.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:36:18.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:36:18.915 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:36:18.915 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:36:18.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:36:18.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:36:18.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:36:19.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:36:19.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:36:19.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:36:19.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:36:19.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:36:19.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:36:19.460 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:36:19.460 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:36:19.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:36:19.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:36:19.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:36:19.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836503 2024-09-18T15:36:19.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836503 2024-09-18T15:36:19.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836503' 2024-09-18T15:36:19.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:36:19.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:36:19.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672984 2024-09-18T15:36:19.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672984 2024-09-18T15:36:19.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836503 1-42949672984' 2024-09-18T15:36:19.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:36:19.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:36:19.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509455 2024-09-18T15:36:19.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509455 2024-09-18T15:36:19.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836503 1-42949672984 2-64424509455' 2024-09-18T15:36:19.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:36:19.939 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836503 2024-09-18T15:36:19.939 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:36:19.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:36:19.942 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836503 2024-09-18T15:36:19.943 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:36:19.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836503 2024-09-18T15:36:19.944 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836503 2024-09-18T15:36:19.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-18T15:36:19.945 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:36:20.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836502 -lt 21474836503 2024-09-18T15:36:20.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:36:21.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:36:21.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:36:21.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836503 -lt 21474836503 2024-09-18T15:36:21.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:36:21.569 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672984 2024-09-18T15:36:21.570 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:36:21.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:36:21.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672984 2024-09-18T15:36:21.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:36:21.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672984 2024-09-18T15:36:21.576 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672984 2024-09-18T15:36:21.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-09-18T15:36:21.576 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:36:21.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672984 -lt 42949672984 2024-09-18T15:36:21.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:36:21.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509455 2024-09-18T15:36:21.914 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:36:21.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:36:21.917 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509455 2024-09-18T15:36:21.918 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:36:21.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509455 2024-09-18T15:36:21.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-18T15:36:21.919 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509455 2024-09-18T15:36:21.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:36:22.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-18T15:36:22.269 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:36:22.269 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:36:22.270 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:36:22.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:36:22.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:36:22.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:36:22.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:36:22.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:36:22.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:36:22.644 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:36:22.644 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:36:22.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:36:22.979 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:36:22.980 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:36:22.980 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:36:23.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:36:23.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:36:23.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:36:23.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: seq 1 7 2024-09-18T15:36:23.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-18T15:36:23.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objname=EOBJ1 2024-09-18T15:36:23.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-09-18T15:36:23.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:36:23.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2024-09-18T15:36:23.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ1 2024-09-18T15:36:23.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:36:23.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:36:23.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:36:23.814 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:36:23.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:36:24.341 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:36:24.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:36:24.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:36:24.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:36:24.614 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: expr 1 % 2 2024-09-18T15:36:24.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: local osd=1 2024-09-18T15:36:24.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: case $i in 2024-09-18T15:36:24.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-09-18T15:36:24.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: echo UVWXYZZZ 2024-09-18T15:36:24.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:36:24.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:36:24.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:36:24.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:36:24.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:36:24.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:36:24.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:36:24.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:36:24.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:36:24.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:36:24.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:36:24.619 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:36:24.619 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:36:24.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:36:24.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:36:24.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:36:25.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:36:25.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:36:25.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:36:25.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:36:25.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:36:25.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:36:25.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:36:25.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:36:28.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:36:28.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:36:28.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:36:28.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:36:28.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:36:28.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:36:28.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:36:28.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:36:28.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:36:28.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:36:28.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:36:28.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:36:28.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:36:28.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:36:28.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:36:28.033 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:36:28.033 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:36:28.033 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:36:28.033 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:36:28.033 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:36:28.033 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:36:28.034 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:36:28.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:36:28.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:36:28.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:36:28.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:36:28.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:36:28.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:36:28.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:36:28.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:36:28.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:36:28.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:36:28.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:36:28.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:36:28.035 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:36:28.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:36:28.037 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:36:28.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:36:28.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:36:28.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:36:28.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:36:28.065 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:36:28.060+0000 7fd8f7f12540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:36:28.069 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:36:28.064+0000 7fd8f7f12540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:36:28.071 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:36:28.064+0000 7fd8f7f12540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:36:28.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:36:28.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:36:28.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:36:28.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:36:28.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:36:28.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:28.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:36:28.349 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:36:28.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:28.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:36:28.606 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:36:28.600+0000 7fd8f7f12540 -1 Falling back to public interface 2024-09-18T15:36:28.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:29.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:29.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:29.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:36:29.676 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:36:29.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:29.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:36:30.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:31.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:31.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:31.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:36:31.013 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:36:31.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:31.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:36:31.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:31.530 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:36:31.524+0000 7fd8f7f12540 -1 osd.1 36 log_to_monitors true 2024-09-18T15:36:32.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:32.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:32.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:36:32.380 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:36:32.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:32.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:36:32.708 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 37 last_clean_interval [10,36) [v2:127.0.0.1:6810/341422,v1:127.0.0.1:6811/341422] [v2:127.0.0.1:6812/341422,v1:127.0.0.1:6813/341422] exists,up 144c133f-b1d4-4eed-ab4d-dcfcf7bf1077 2024-09-18T15:36:32.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:36:32.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:36:32.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:36:32.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:36:32.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:36:32.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:36:32.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:36:32.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:36:32.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:36:32.710 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:36:32.710 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:36:32.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:36:32.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:36:32.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:36:32.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:36:32.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:36:32.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:36:32.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:36:32.919 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:36:33.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:36:33.433 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:36:33.433 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:36:33.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:36:33.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:36:33.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:36:33.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836507 2024-09-18T15:36:33.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836507 2024-09-18T15:36:33.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507' 2024-09-18T15:36:33.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:36:33.665 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:36:33.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=176093659138 2024-09-18T15:36:33.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 176093659138 2024-09-18T15:36:33.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507 1-176093659138' 2024-09-18T15:36:33.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:36:33.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:36:34.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509459 2024-09-18T15:36:34.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509459 2024-09-18T15:36:34.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507 1-176093659138 2-64424509459' 2024-09-18T15:36:34.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:36:34.067 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836507 2024-09-18T15:36:34.068 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:36:34.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:36:34.071 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836507 2024-09-18T15:36:34.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:36:34.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836507 2024-09-18T15:36:34.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-18T15:36:34.074 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836507 2024-09-18T15:36:34.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:36:34.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836505 -lt 21474836507 2024-09-18T15:36:34.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:36:35.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:36:35.392 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:36:35.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836507 -lt 21474836507 2024-09-18T15:36:35.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:36:35.712 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-176093659138 2024-09-18T15:36:35.712 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:36:35.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:36:35.715 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-176093659138 2024-09-18T15:36:35.716 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:36:35.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=176093659138 2024-09-18T15:36:35.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 176093659138' 2024-09-18T15:36:35.721 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 176093659138 2024-09-18T15:36:35.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:36:36.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 176093659138 -lt 176093659138 2024-09-18T15:36:36.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:36:36.067 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509459 2024-09-18T15:36:36.067 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:36:36.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:36:36.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509459 2024-09-18T15:36:36.071 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:36:36.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509459 2024-09-18T15:36:36.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-18T15:36:36.072 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509459 2024-09-18T15:36:36.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:36:36.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509459 -lt 64424509459 2024-09-18T15:36:36.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:36:36.415 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:36:36.415 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:36:36.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:36:36.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:36:36.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:36:36.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:36:36.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:36:36.779 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:36:36.779 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:36:36.779 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:36:37.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:36:37.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:36:37.124 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:36:37.124 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:36:37.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-09-18T15:36:37.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-09-18T15:36:37.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T15:36:37.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-09-18T15:36:37.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:36:37.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:36:37.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:36:37.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:36:37.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:36:37.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:36:37.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:36:37.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:36:37.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:36:37.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:36:37.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:36:37.909 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:36:37.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:36:37.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:36:38.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-09-18T15:36:38.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-09-18T15:36:38.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:36:38.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:36:38.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:36:38.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:36:38.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:36:38.278 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:36:38.278 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:36:38.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T15:36:38.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T15:36:38.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-09-18T15:36:38.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:36:38.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-18T15:36:38.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:36:38.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:36:38.895 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:36:38.895 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:36:38.895 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:36:38.895 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:36:38.896 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:36:38.896 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:36:39.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:36:39.204 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:36:39.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:36:39.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:36:39.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:36:39.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:36:39.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:36:39.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-18T15:36:39.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objname=EOBJ2 2024-09-18T15:36:39.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-09-18T15:36:39.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:36:39.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2024-09-18T15:36:39.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ2 2024-09-18T15:36:39.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:36:39.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:36:39.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:36:40.026 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:36:40.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:36:40.512 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:36:40.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:36:40.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:36:40.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:36:40.776 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: expr 2 % 2 2024-09-18T15:36:40.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: local osd=0 2024-09-18T15:36:40.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: case $i in 2024-09-18T15:36:40.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3649: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-09-18T15:36:40.780 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records in 2024-09-18T15:36:40.780 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records out 2024-09-18T15:36:40.780 INFO:tasks.workunit.client.0.smithi046.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 0.000250984 s, 8.2 MB/s 2024-09-18T15:36:40.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3650: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:36:40.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:36:40.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:36:40.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:36:40.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:36:40.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:36:40.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:36:40.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:36:40.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:36:40.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:36:40.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:36:40.782 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:36:40.783 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:36:40.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:36:40.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:36:40.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:36:41.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:36:41.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:36:41.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:36:41.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:36:41.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:36:41.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:36:41.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:36:41.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:36:43.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-18T15:36:43.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:36:43.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:36:43.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:36:43.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:36:43.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:36:43.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:36:43.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:36:43.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:36:43.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:36:43.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:36:43.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:36:43.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:36:43.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:36:43.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:36:43.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:36:43.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:36:43.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:36:43.965 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:36:43.965 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:36:43.965 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:36:43.966 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:36:43.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:36:43.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:36:43.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:36:43.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:36:43.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:36:43.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:36:43.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:36:43.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:36:43.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:36:43.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:36:43.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:36:43.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:36:43.968 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:36:43.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:36:43.970 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:36:43.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:36:43.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:36:43.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:36:43.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:36:44.000 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:36:43.995+0000 7f9ba6427540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:36:44.003 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:36:43.995+0000 7f9ba6427540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:36:44.005 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:36:43.999+0000 7f9ba6427540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:36:44.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:36:44.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:36:44.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:36:44.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:36:44.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:36:44.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:44.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:36:44.275 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:36:44.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:44.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:36:44.562 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:36:44.555+0000 7f9ba6427540 -1 Falling back to public interface 2024-09-18T15:36:44.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:45.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:45.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:45.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:36:45.620 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:36:45.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:45.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:36:45.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:46.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:46.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:46.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:36:46.940 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:36:46.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:46.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:36:47.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:47.372 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:36:47.367+0000 7f9ba6427540 -1 osd.0 46 log_to_monitors true 2024-09-18T15:36:48.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:48.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:48.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:36:48.348 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:36:48.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:48.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:36:48.705 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 51 up_thru 51 down_at 47 last_clean_interval [5,46) [v2:127.0.0.1:6802/342971,v1:127.0.0.1:6803/342971] [v2:127.0.0.1:6804/342971,v1:127.0.0.1:6805/342971] exists,up d366d8b9-5243-44c9-8553-5749a6e3a9c7 2024-09-18T15:36:48.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:36:48.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:36:48.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:36:48.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:36:48.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:36:48.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:36:48.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:36:48.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:36:48.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:36:48.707 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:36:48.707 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:36:48.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:36:48.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:36:48.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:36:48.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:36:48.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:36:48.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:36:48.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:36:48.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:36:49.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:36:49.447 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:36:49.447 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:36:49.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:36:49.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:36:49.448 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:36:49.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=219043332098 2024-09-18T15:36:49.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 219043332098 2024-09-18T15:36:49.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-219043332098' 2024-09-18T15:36:49.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:36:49.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:36:49.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=176093659142 2024-09-18T15:36:49.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 176093659142 2024-09-18T15:36:49.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-219043332098 1-176093659142' 2024-09-18T15:36:49.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:36:49.775 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:36:49.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509463 2024-09-18T15:36:49.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509463 2024-09-18T15:36:49.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-219043332098 1-176093659142 2-64424509463' 2024-09-18T15:36:49.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:36:49.918 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-219043332098 2024-09-18T15:36:49.918 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:36:49.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:36:49.922 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-219043332098 2024-09-18T15:36:49.922 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:36:49.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=219043332098 2024-09-18T15:36:49.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 219043332098' 2024-09-18T15:36:49.924 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 219043332098 2024-09-18T15:36:49.924 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:36:50.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 219043332098 2024-09-18T15:36:50.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:36:51.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:36:51.248 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:36:51.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 219043332098 -lt 219043332098 2024-09-18T15:36:51.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:36:51.568 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-176093659142 2024-09-18T15:36:51.568 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:36:51.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:36:51.572 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-176093659142 2024-09-18T15:36:51.572 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:36:51.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=176093659142 2024-09-18T15:36:51.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 176093659142' 2024-09-18T15:36:51.574 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 176093659142 2024-09-18T15:36:51.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:36:51.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 176093659142 -lt 176093659142 2024-09-18T15:36:51.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:36:51.909 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509463 2024-09-18T15:36:51.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:36:51.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:36:51.914 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509463 2024-09-18T15:36:51.914 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:36:51.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509463 2024-09-18T15:36:51.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-09-18T15:36:51.916 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509463 2024-09-18T15:36:51.917 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:36:52.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509463 -lt 64424509463 2024-09-18T15:36:52.259 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:36:52.260 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:36:52.260 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:36:52.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:36:52.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:36:52.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:36:52.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:36:52.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:36:52.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:36:52.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:36:52.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:36:52.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:36:52.926 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:36:52.927 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:36:52.927 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:36:53.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:36:53.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:36:53.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:36:53.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-18T15:36:53.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objname=EOBJ3 2024-09-18T15:36:53.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-09-18T15:36:53.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:36:53.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2024-09-18T15:36:53.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ3 2024-09-18T15:36:53.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:36:53.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:36:53.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:36:53.729 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:36:53.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:36:54.230 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:36:54.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:36:54.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:36:54.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:36:54.529 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: expr 3 % 2 2024-09-18T15:36:54.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: local osd=1 2024-09-18T15:36:54.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: case $i in 2024-09-18T15:36:54.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3655: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-09-18T15:36:54.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:36:54.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:36:54.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:36:54.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:36:54.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-09-18T15:36:54.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:36:54.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:36:54.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:36:54.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:36:54.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:36:54.533 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:36:54.533 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:36:54.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:36:54.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:36:54.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:36:54.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:36:54.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-09-18T15:36:54.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:36:54.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:36:54.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:36:54.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:36:54.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:36:54.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-09-18T15:36:56.836 INFO:tasks.workunit.client.0.smithi046.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-09-18T15:36:57.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:36:57.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:36:57.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:36:57.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:36:57.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:36:57.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:36:57.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:36:57.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:36:57.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:36:57.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:36:57.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:36:57.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:36:57.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:36:57.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:36:57.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:36:57.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:36:57.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:36:57.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:36:57.373 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:36:57.373 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:36:57.373 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:36:57.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:36:57.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:36:57.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:36:57.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:36:57.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:36:57.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:36:57.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:36:57.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:36:57.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:36:57.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:36:57.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:36:57.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:36:57.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:36:57.379 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:36:57.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:36:57.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:36:57.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:36:57.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:36:57.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:36:57.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:36:57.408 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:36:57.399+0000 7fd973dc9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:36:57.411 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:36:57.403+0000 7fd973dc9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:36:57.412 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:36:57.407+0000 7fd973dc9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:36:57.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:36:57.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:36:57.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:36:57.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:36:57.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:36:57.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:57.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:36:57.691 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:36:57.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:57.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:36:57.954 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:36:57.947+0000 7fd973dc9540 -1 Falling back to public interface 2024-09-18T15:36:58.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:59.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:59.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:59.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:36:59.029 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:36:59.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:59.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:36:59.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:37:00.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:37:00.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:37:00.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:37:00.388 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:37:00.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:37:00.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:37:00.679 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:37:00.671+0000 7fd973dc9540 -1 osd.1 55 log_to_monitors true 2024-09-18T15:37:00.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:37:01.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:37:01.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:37:01.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:37:01.721 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:37:01.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:37:01.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:37:02.217 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 60 up_thru 60 down_at 56 last_clean_interval [41,55) [v2:127.0.0.1:6810/344392,v1:127.0.0.1:6811/344392] [v2:127.0.0.1:6812/344392,v1:127.0.0.1:6813/344392] exists,up 144c133f-b1d4-4eed-ab4d-dcfcf7bf1077 2024-09-18T15:37:02.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:37:02.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:37:02.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:37:02.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:37:02.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:37:02.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:37:02.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:37:02.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:37:02.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:37:02.219 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:37:02.219 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:37:02.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:37:02.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:37:02.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:37:02.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:37:02.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:37:02.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:37:02.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:37:02.439 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:37:02.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:37:02.777 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:37:02.777 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:37:02.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:37:02.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:37:02.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:37:02.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=219043332102 2024-09-18T15:37:02.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 219043332102 2024-09-18T15:37:02.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-219043332102' 2024-09-18T15:37:02.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:37:02.917 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:37:03.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=257698037762 2024-09-18T15:37:03.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 257698037762 2024-09-18T15:37:03.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-219043332102 1-257698037762' 2024-09-18T15:37:03.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:37:03.103 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:37:03.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509467 2024-09-18T15:37:03.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509467 2024-09-18T15:37:03.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-219043332102 1-257698037762 2-64424509467' 2024-09-18T15:37:03.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:37:03.292 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-219043332102 2024-09-18T15:37:03.292 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:37:03.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:37:03.296 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-219043332102 2024-09-18T15:37:03.296 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:37:03.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=219043332102 2024-09-18T15:37:03.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 219043332102' 2024-09-18T15:37:03.298 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 219043332102 2024-09-18T15:37:03.298 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:37:03.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 219043332100 -lt 219043332102 2024-09-18T15:37:03.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:37:04.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:37:04.614 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:37:04.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 219043332102 -lt 219043332102 2024-09-18T15:37:04.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:37:04.941 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-257698037762 2024-09-18T15:37:04.941 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:37:04.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:37:04.945 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-257698037762 2024-09-18T15:37:04.945 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:37:04.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=257698037762 2024-09-18T15:37:04.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 257698037762' 2024-09-18T15:37:04.948 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 257698037762 2024-09-18T15:37:04.948 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:37:05.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 257698037762 -lt 257698037762 2024-09-18T15:37:05.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:37:05.277 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509467 2024-09-18T15:37:05.277 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:37:05.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:37:05.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509467 2024-09-18T15:37:05.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:37:05.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509467 2024-09-18T15:37:05.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-09-18T15:37:05.283 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509467 2024-09-18T15:37:05.283 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:37:05.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509467 -lt 64424509467 2024-09-18T15:37:05.625 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:37:05.625 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:05.626 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:06.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:37:06.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:06.033 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:06.033 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:06.033 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:06.033 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:06.034 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:06.034 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:06.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:37:06.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:06.369 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:06.369 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:06.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:37:06.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:37:06.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:37:06.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-18T15:37:06.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objname=EOBJ4 2024-09-18T15:37:06.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-09-18T15:37:06.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:37:06.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2024-09-18T15:37:06.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ4 2024-09-18T15:37:06.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:37:06.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:37:06.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:37:07.220 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:37:07.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:37:07.722 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:37:07.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:37:07.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:37:07.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:37:07.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: expr 4 % 2 2024-09-18T15:37:07.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: local osd=0 2024-09-18T15:37:07.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: case $i in 2024-09-18T15:37:07.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-09-18T15:37:08.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3660: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-09-18T15:37:08.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: echo -n bad-val 2024-09-18T15:37:08.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-18T15:37:08.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:37:08.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:37:08.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:37:08.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:37:08.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-18T15:37:08.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:37:08.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:37:08.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:37:08.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:37:08.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:37:08.344 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:37:08.344 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:37:08.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:37:08.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:37:08.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:37:08.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:37:08.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-18T15:37:08.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:37:08.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:37:08.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:37:08.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:37:08.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:37:08.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-18T15:37:11.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-18T15:37:11.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:37:11.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:37:11.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:37:11.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:37:11.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:37:11.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:37:11.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:37:11.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:37:11.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:37:11.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:37:11.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:37:11.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:37:11.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:37:11.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:37:11.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:37:11.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:37:11.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:37:11.209 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:37:11.209 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:37:11.209 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:37:11.211 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:37:11.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:37:11.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:37:11.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:37:11.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:37:11.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:37:11.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:37:11.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:37:11.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:37:11.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:37:11.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:37:11.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:37:11.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:37:11.213 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:37:11.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:37:11.214 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:37:11.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:37:11.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:37:11.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:37:11.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:37:11.245 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:37:11.239+0000 7f0a7c7aa540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:37:11.252 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:37:11.243+0000 7f0a7c7aa540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:37:11.253 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:37:11.243+0000 7f0a7c7aa540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:37:11.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:37:11.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:37:11.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:37:11.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:37:11.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:37:11.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:37:11.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:37:11.524 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:37:11.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:37:11.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:37:11.786 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:37:11.779+0000 7f0a7c7aa540 -1 Falling back to public interface 2024-09-18T15:37:11.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:37:12.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:37:12.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:37:12.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:37:12.824 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:37:12.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:37:12.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:37:13.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:37:14.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:37:14.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:37:14.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:37:14.133 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:37:14.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:37:14.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:37:14.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:37:14.600 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:37:14.591+0000 7f0a7c7aa540 -1 osd.0 64 log_to_monitors true 2024-09-18T15:37:15.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:37:15.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:37:15.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:37:15.502 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:37:15.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:37:15.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:37:15.910 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 69 up_thru 69 down_at 65 last_clean_interval [51,64) [v2:127.0.0.1:6802/345856,v1:127.0.0.1:6803/345856] [v2:127.0.0.1:6804/345856,v1:127.0.0.1:6805/345856] exists,up d366d8b9-5243-44c9-8553-5749a6e3a9c7 2024-09-18T15:37:15.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:37:15.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:37:15.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:37:15.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:37:15.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:37:15.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:37:15.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:37:15.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:37:15.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:37:15.912 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:37:15.912 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:37:15.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:37:15.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:37:15.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:37:16.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:37:16.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:37:16.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:37:16.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:37:16.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:37:16.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:37:16.513 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:37:16.513 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:37:16.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:37:16.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:37:16.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:37:16.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=296352743426 2024-09-18T15:37:16.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 296352743426 2024-09-18T15:37:16.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-296352743426' 2024-09-18T15:37:16.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:37:16.653 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:37:16.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=257698037766 2024-09-18T15:37:16.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 257698037766 2024-09-18T15:37:16.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-296352743426 1-257698037766' 2024-09-18T15:37:16.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:37:16.914 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:37:17.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509471 2024-09-18T15:37:17.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509471 2024-09-18T15:37:17.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-296352743426 1-257698037766 2-64424509471' 2024-09-18T15:37:17.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:37:17.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-296352743426 2024-09-18T15:37:17.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:37:17.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:37:17.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-296352743426 2024-09-18T15:37:17.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:37:17.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=296352743426 2024-09-18T15:37:17.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 296352743426' 2024-09-18T15:37:17.065 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 296352743426 2024-09-18T15:37:17.065 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:37:17.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 296352743426 2024-09-18T15:37:17.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:37:18.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:37:18.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:37:18.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 296352743426 -lt 296352743426 2024-09-18T15:37:18.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:37:18.696 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-257698037766 2024-09-18T15:37:18.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:37:18.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:37:18.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-257698037766 2024-09-18T15:37:18.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:37:18.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=257698037766 2024-09-18T15:37:18.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 257698037766' 2024-09-18T15:37:18.702 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 257698037766 2024-09-18T15:37:18.703 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:37:19.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 257698037766 -lt 257698037766 2024-09-18T15:37:19.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:37:19.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509471 2024-09-18T15:37:19.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:37:19.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:37:19.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509471 2024-09-18T15:37:19.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:37:19.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509471 2024-09-18T15:37:19.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-09-18T15:37:19.064 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509471 2024-09-18T15:37:19.065 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:37:19.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509471 -lt 64424509471 2024-09-18T15:37:19.401 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:37:19.401 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:19.402 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:19.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:37:19.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:19.810 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:19.810 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:19.810 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:19.810 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:19.811 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:19.811 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:20.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:37:20.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:20.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:20.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:20.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:37:20.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:37:20.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:37:20.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-18T15:37:20.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:37:20.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:37:20.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-18T15:37:20.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:37:20.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-18T15:37:20.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:37:20.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:37:20.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-18T15:37:20.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:37:20.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-18T15:37:20.521 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:37:20.521 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:37:20.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:37:20.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:37:20.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:37:21.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:37:21.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-18T15:37:21.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:37:21.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:37:21.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-18T15:37:21.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:37:21.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:37:21.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-18T15:37:23.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-18T15:37:23.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:37:23.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:37:23.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T15:37:23.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:37:23.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:37:23.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:37:23.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:37:23.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:37:23.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:37:23.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T15:37:23.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T15:37:23.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:37:23.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:37:23.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:37:23.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:37:23.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:37:23.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:37:23.821 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:37:23.821 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:37:23.821 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:37:23.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:37:23.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:37:23.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:37:23.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:37:23.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:37:23.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:37:23.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:37:23.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:37:23.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:37:23.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:37:23.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:37:23.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-18T15:37:23.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T15:37:23.824 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.2 2024-09-18T15:37:23.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:37:23.825 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-18T15:37:23.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T15:37:23.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:37:23.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:37:23.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:37:23.858 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:37:23.851+0000 7f9317692540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:37:23.862 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:37:23.855+0000 7f9317692540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:37:23.864 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:37:23.859+0000 7f9317692540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:37:24.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T15:37:24.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:37:24.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:37:24.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:37:24.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:37:24.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:37:24.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:37:24.145 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:37:24.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:37:24.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:37:24.402 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:37:24.395+0000 7f9317692540 -1 Falling back to public interface 2024-09-18T15:37:24.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:37:25.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:37:25.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:37:25.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:37:25.505 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:37:25.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:37:25.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:37:25.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:37:26.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:37:26.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:37:26.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:37:26.823 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:37:26.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:37:26.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:37:27.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:37:27.225 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:37:27.219+0000 7f9317692540 -1 osd.2 71 log_to_monitors true 2024-09-18T15:37:28.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:37:28.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:37:28.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:37:28.134 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:37:28.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:37:28.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:37:28.535 INFO:tasks.workunit.client.0.smithi046.stderr:osd.2 up in weight 1 up_from 76 up_thru 76 down_at 72 last_clean_interval [15,71) [v2:127.0.0.1:6818/347211,v1:127.0.0.1:6819/347211] [v2:127.0.0.1:6820/347211,v1:127.0.0.1:6821/347211] exists,up 1b922d60-8949-4a5a-9a47-86dba5eb7687 2024-09-18T15:37:28.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:37:28.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:37:28.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:37:28.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:37:28.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:37:28.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:37:28.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:37:28.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:37:28.536 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:37:28.537 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:37:28.537 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:37:28.537 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:37:28.537 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:37:28.537 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:37:28.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:37:28.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:37:28.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:37:28.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:37:28.767 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:37:29.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:37:29.211 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:37:29.211 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:37:29.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:37:29.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:37:29.211 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:37:29.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=296352743429 2024-09-18T15:37:29.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 296352743429 2024-09-18T15:37:29.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-296352743429' 2024-09-18T15:37:29.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:37:29.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:37:29.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=257698037769 2024-09-18T15:37:29.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 257698037769 2024-09-18T15:37:29.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-296352743429 1-257698037769' 2024-09-18T15:37:29.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:37:29.503 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:37:29.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=326417514498 2024-09-18T15:37:29.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 326417514498 2024-09-18T15:37:29.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-296352743429 1-257698037769 2-326417514498' 2024-09-18T15:37:29.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:37:29.652 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-296352743429 2024-09-18T15:37:29.653 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:37:29.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:37:29.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-296352743429 2024-09-18T15:37:29.657 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:37:29.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=296352743429 2024-09-18T15:37:29.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 296352743429' 2024-09-18T15:37:29.659 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 296352743429 2024-09-18T15:37:29.659 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:37:29.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 296352743428 -lt 296352743429 2024-09-18T15:37:29.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:37:30.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:37:30.975 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:37:31.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 296352743430 -lt 296352743429 2024-09-18T15:37:31.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:37:31.289 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-257698037769 2024-09-18T15:37:31.289 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:37:31.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:37:31.292 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-257698037769 2024-09-18T15:37:31.293 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:37:31.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=257698037769 2024-09-18T15:37:31.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 257698037769' 2024-09-18T15:37:31.295 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 257698037769 2024-09-18T15:37:31.296 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:37:31.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 257698037769 -lt 257698037769 2024-09-18T15:37:31.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:37:31.601 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-326417514498 2024-09-18T15:37:31.601 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:37:31.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:37:31.604 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-326417514498 2024-09-18T15:37:31.604 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:37:31.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=326417514498 2024-09-18T15:37:31.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 326417514498' 2024-09-18T15:37:31.607 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 326417514498 2024-09-18T15:37:31.607 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:37:31.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 326417514498 -lt 326417514498 2024-09-18T15:37:31.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:37:31.956 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:31.956 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:32.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:37:32.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:32.364 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:32.365 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:32.365 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:32.365 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:32.365 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:32.365 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:32.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:37:32.735 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:32.735 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:32.735 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:33.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:37:33.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:37:33.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:37:33.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3666: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-09-18T15:37:33.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3667: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-18T15:37:33.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:37:33.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:37:33.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-18T15:37:33.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:37:33.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-18T15:37:33.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:37:33.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:37:33.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-18T15:37:33.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:37:33.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-18T15:37:33.102 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:37:33.102 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:37:33.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:37:33.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:37:33.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:37:33.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:37:33.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-18T15:37:33.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:37:33.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:37:33.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-18T15:37:33.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:37:33.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:37:33.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-18T15:37:35.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-18T15:37:35.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:37:35.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:37:35.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T15:37:35.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:37:35.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:37:35.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:37:35.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:37:35.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:37:35.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:37:35.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T15:37:35.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T15:37:35.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:37:35.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:37:35.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:37:35.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:37:35.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:37:35.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:37:35.996 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:37:35.996 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:37:35.996 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:37:35.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:37:35.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:37:35.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:37:35.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:37:35.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:37:35.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:37:35.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:37:35.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:37:35.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:37:35.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:37:35.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:37:35.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-18T15:37:35.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T15:37:35.999 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.2 2024-09-18T15:37:36.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:37:36.001 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-18T15:37:36.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T15:37:36.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:37:36.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:37:36.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:37:36.032 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:37:36.027+0000 7f59e0f34540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:37:36.036 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:37:36.031+0000 7f59e0f34540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:37:36.048 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:37:36.031+0000 7f59e0f34540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:37:36.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T15:37:36.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:37:36.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:37:36.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:37:36.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:37:36.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:37:36.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:37:36.305 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:37:36.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:37:36.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:37:36.570 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:37:36.563+0000 7f59e0f34540 -1 Falling back to public interface 2024-09-18T15:37:36.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:37:37.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:37:37.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:37:37.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:37:37.682 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:37:37.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:37:37.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:37:37.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:37:39.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:37:39.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:37:39.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:37:39.001 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:37:39.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:37:39.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:37:39.032 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:37:39.027+0000 7f59e0f34540 -1 osd.2 78 log_to_monitors true 2024-09-18T15:37:39.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:37:40.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:37:40.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:37:40.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:37:40.434 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:37:40.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:37:40.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:37:40.748 INFO:tasks.workunit.client.0.smithi046.stderr:osd.2 up in weight 1 up_from 83 up_thru 83 down_at 79 last_clean_interval [76,78) [v2:127.0.0.1:6818/348565,v1:127.0.0.1:6819/348565] [v2:127.0.0.1:6820/348565,v1:127.0.0.1:6821/348565] exists,up 1b922d60-8949-4a5a-9a47-86dba5eb7687 2024-09-18T15:37:40.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:37:40.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:37:40.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:37:40.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:37:40.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:37:40.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:37:40.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:37:40.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:37:40.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:37:40.749 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:37:40.749 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:37:40.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:37:40.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:37:40.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:37:40.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:37:40.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:37:40.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:37:40.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:37:40.978 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:37:41.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:37:41.294 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:37:41.294 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:37:41.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:37:41.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:37:41.294 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:37:41.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=296352743433 2024-09-18T15:37:41.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 296352743433 2024-09-18T15:37:41.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-296352743433' 2024-09-18T15:37:41.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:37:41.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:37:41.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=257698037773 2024-09-18T15:37:41.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 257698037773 2024-09-18T15:37:41.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-296352743433 1-257698037773' 2024-09-18T15:37:41.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:37:41.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:37:41.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=356482285570 2024-09-18T15:37:41.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 356482285570 2024-09-18T15:37:41.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-296352743433 1-257698037773 2-356482285570' 2024-09-18T15:37:41.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:37:41.716 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-296352743433 2024-09-18T15:37:41.716 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:37:41.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:37:41.719 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-296352743433 2024-09-18T15:37:41.719 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:37:41.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=296352743433 2024-09-18T15:37:41.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 296352743433' 2024-09-18T15:37:41.721 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 296352743433 2024-09-18T15:37:41.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:37:42.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 296352743432 -lt 296352743433 2024-09-18T15:37:42.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:37:43.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:37:43.034 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:37:43.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 296352743433 -lt 296352743433 2024-09-18T15:37:43.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:37:43.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-257698037773 2024-09-18T15:37:43.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:37:43.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:37:43.390 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-257698037773 2024-09-18T15:37:43.390 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:37:43.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=257698037773 2024-09-18T15:37:43.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 257698037773' 2024-09-18T15:37:43.393 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 257698037773 2024-09-18T15:37:43.394 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:37:43.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 257698037773 -lt 257698037773 2024-09-18T15:37:43.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:37:43.732 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-356482285570 2024-09-18T15:37:43.732 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:37:43.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:37:43.735 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-356482285570 2024-09-18T15:37:43.735 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:37:43.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=356482285570 2024-09-18T15:37:43.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 356482285570' 2024-09-18T15:37:43.737 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 356482285570 2024-09-18T15:37:43.738 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:37:44.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 356482285570 -lt 356482285570 2024-09-18T15:37:44.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:37:44.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:44.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:44.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:37:44.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:44.494 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:44.494 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:44.494 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:44.494 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:44.494 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:44.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:44.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:37:44.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:44.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:44.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:45.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:37:45.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:37:45.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:37:45.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-09-18T15:37:45.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-18T15:37:45.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objname=EOBJ5 2024-09-18T15:37:45.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-09-18T15:37:45.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:37:45.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2024-09-18T15:37:45.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ5 2024-09-18T15:37:45.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:37:45.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:37:45.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:37:45.688 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:37:45.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:37:46.188 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:37:46.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:37:46.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:37:46.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:37:46.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: expr 5 % 2 2024-09-18T15:37:46.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: local osd=1 2024-09-18T15:37:46.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: case $i in 2024-09-18T15:37:46.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3673: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-09-18T15:37:46.601 INFO:tasks.workunit.client.0.smithi046.stderr:2+0 records in 2024-09-18T15:37:46.601 INFO:tasks.workunit.client.0.smithi046.stderr:2+0 records out 2024-09-18T15:37:46.601 INFO:tasks.workunit.client.0.smithi046.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000265983 s, 15.4 MB/s 2024-09-18T15:37:46.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3674: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:37:46.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:37:46.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:37:46.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:37:46.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:37:46.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:37:46.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:37:46.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:37:46.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:37:46.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:37:46.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:37:46.604 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:37:46.604 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:37:46.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:37:46.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:37:46.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:37:46.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:37:46.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:37:46.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:37:46.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:37:46.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:37:46.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:37:46.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:37:46.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:37:49.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:37:49.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:37:49.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:37:49.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:37:49.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:37:49.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:37:49.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:37:49.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:37:49.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:37:49.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:37:49.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:37:49.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:37:49.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:37:49.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:37:49.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:37:49.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:37:49.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:37:49.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:37:49.453 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:37:49.453 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:37:49.454 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:37:49.454 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:37:49.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:37:49.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:37:49.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:37:49.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:37:49.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:37:49.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:37:49.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:37:49.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:37:49.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:37:49.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:37:49.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:37:49.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:37:49.456 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:37:49.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:37:49.457 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:37:49.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:37:49.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:37:49.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:37:49.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:37:49.487 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:37:49.479+0000 7f09d3d52540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:37:49.490 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:37:49.483+0000 7f09d3d52540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:37:49.496 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:37:49.487+0000 7f09d3d52540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:37:49.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:37:49.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:37:49.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:37:49.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:37:49.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:37:49.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:37:49.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:37:49.762 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:37:49.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:37:49.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:37:50.058 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:37:50.051+0000 7f09d3d52540 -1 Falling back to public interface 2024-09-18T15:37:50.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:37:51.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:37:51.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:37:51.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:37:51.117 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:37:51.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:37:51.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:37:51.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:37:52.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:37:52.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:37:52.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:37:52.458 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:37:52.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:37:52.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:37:52.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:37:52.999 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:37:52.991+0000 7f09d3d52540 -1 osd.1 87 log_to_monitors true 2024-09-18T15:37:53.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:37:53.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:37:53.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:37:53.769 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:37:53.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:37:53.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:37:54.230 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 92 up_thru 83 down_at 88 last_clean_interval [60,87) [v2:127.0.0.1:6810/349988,v1:127.0.0.1:6811/349988] [v2:127.0.0.1:6812/349988,v1:127.0.0.1:6813/349988] exists,up 144c133f-b1d4-4eed-ab4d-dcfcf7bf1077 2024-09-18T15:37:54.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:37:54.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:37:54.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:37:54.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:37:54.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:37:54.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:37:54.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:37:54.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:37:54.231 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:37:54.232 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:37:54.232 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:37:54.233 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:37:54.233 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:37:54.233 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:37:54.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:37:54.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:37:54.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:37:54.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:37:54.456 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:37:54.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:37:54.874 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:37:54.874 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:37:54.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:37:54.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:37:54.875 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:37:55.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=296352743437 2024-09-18T15:37:55.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 296352743437 2024-09-18T15:37:55.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-296352743437' 2024-09-18T15:37:55.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:37:55.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:37:55.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=395136991234 2024-09-18T15:37:55.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 395136991234 2024-09-18T15:37:55.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-296352743437 1-395136991234' 2024-09-18T15:37:55.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:37:55.298 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:37:55.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=356482285574 2024-09-18T15:37:55.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 356482285574 2024-09-18T15:37:55.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-296352743437 1-395136991234 2-356482285574' 2024-09-18T15:37:55.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:37:55.450 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-296352743437 2024-09-18T15:37:55.450 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:37:55.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:37:55.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-296352743437 2024-09-18T15:37:55.454 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:37:55.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=296352743437 2024-09-18T15:37:55.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 296352743437' 2024-09-18T15:37:55.455 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 296352743437 2024-09-18T15:37:55.455 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:37:55.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 296352743435 -lt 296352743437 2024-09-18T15:37:55.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:37:56.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:37:56.767 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:37:57.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 296352743437 -lt 296352743437 2024-09-18T15:37:57.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:37:57.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-395136991234 2024-09-18T15:37:57.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:37:57.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:37:57.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-395136991234 2024-09-18T15:37:57.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:37:57.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=395136991234 2024-09-18T15:37:57.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 395136991234' 2024-09-18T15:37:57.087 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 395136991234 2024-09-18T15:37:57.088 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:37:57.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 395136991234 -lt 395136991234 2024-09-18T15:37:57.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:37:57.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-356482285574 2024-09-18T15:37:57.426 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:37:57.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:37:57.429 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-356482285574 2024-09-18T15:37:57.429 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:37:57.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=356482285574 2024-09-18T15:37:57.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 356482285574' 2024-09-18T15:37:57.431 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 356482285574 2024-09-18T15:37:57.432 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:37:57.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 356482285574 -lt 356482285574 2024-09-18T15:37:57.763 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:37:57.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:57.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:58.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:37:58.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:58.184 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:58.184 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:58.184 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:58.185 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:58.185 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:58.185 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:58.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:37:58.533 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:58.534 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:58.534 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:58.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:37:58.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:37:58.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:37:58.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-18T15:37:58.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objname=EOBJ6 2024-09-18T15:37:58.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-09-18T15:37:58.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:37:58.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2024-09-18T15:37:58.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ6 2024-09-18T15:37:58.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:37:58.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:37:58.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:37:59.384 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:37:59.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:37:59.945 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:37:59.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:37:59.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:37:59.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:38:00.177 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: expr 6 % 2 2024-09-18T15:38:00.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: local osd=0 2024-09-18T15:38:00.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: case $i in 2024-09-18T15:38:00.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-18T15:38:00.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:38:00.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:38:00.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:38:00.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:38:00.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-18T15:38:00.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:38:00.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:38:00.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:38:00.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:38:00.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:38:00.180 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:38:00.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:38:00.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:38:00.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:38:00.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:38:00.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:38:00.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-18T15:38:00.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:38:00.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:38:00.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:38:00.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:38:00.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:38:00.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-09-18T15:38:03.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-18T15:38:03.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:38:03.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:38:03.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:38:03.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:38:03.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:38:03.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:38:03.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:38:03.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:38:03.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:38:03.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:38:03.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:38:03.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:38:03.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:38:03.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:38:03.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:38:03.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:38:03.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:38:03.081 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:38:03.081 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:38:03.081 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:38:03.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:38:03.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:38:03.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:38:03.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:38:03.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:38:03.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:38:03.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:38:03.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:38:03.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:38:03.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:38:03.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:38:03.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:38:03.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:38:03.084 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:38:03.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:38:03.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:38:03.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:38:03.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:38:03.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:38:03.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:38:03.116 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:38:03.110+0000 7f88d677c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:38:03.119 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:38:03.110+0000 7f88d677c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:38:03.124 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:38:03.114+0000 7f88d677c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:38:03.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:38:03.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:38:03.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:38:03.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:38:03.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:38:03.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:03.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:38:03.452 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:38:03.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:03.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:03.695 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:38:03.686+0000 7f88d677c540 -1 Falling back to public interface 2024-09-18T15:38:03.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:04.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:04.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:04.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:38:04.801 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:38:04.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:04.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:05.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:06.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:06.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:06.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:38:06.116 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:38:06.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:06.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:06.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:06.466 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:38:06.458+0000 7f88d677c540 -1 osd.0 96 log_to_monitors true 2024-09-18T15:38:07.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:07.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:07.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:38:07.439 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:38:07.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:07.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:07.753 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 101 up_thru 101 down_at 97 last_clean_interval [69,96) [v2:127.0.0.1:6802/351410,v1:127.0.0.1:6803/351410] [v2:127.0.0.1:6804/351410,v1:127.0.0.1:6805/351410] exists,up d366d8b9-5243-44c9-8553-5749a6e3a9c7 2024-09-18T15:38:07.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:38:07.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:38:07.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:38:07.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:38:07.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:38:07.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:38:07.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:38:07.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:38:07.754 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:38:07.754 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:38:07.754 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:38:07.754 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:38:07.754 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:38:07.754 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:38:07.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:38:07.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:38:07.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:38:07.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:38:07.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:38:08.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:38:08.349 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:38:08.349 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:38:08.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:38:08.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:38:08.350 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:38:08.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=433791696898 2024-09-18T15:38:08.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 433791696898 2024-09-18T15:38:08.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-433791696898' 2024-09-18T15:38:08.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:38:08.551 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:38:08.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=395136991238 2024-09-18T15:38:08.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 395136991238 2024-09-18T15:38:08.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-433791696898 1-395136991238' 2024-09-18T15:38:08.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:38:08.696 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:38:08.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=356482285577 2024-09-18T15:38:08.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 356482285577 2024-09-18T15:38:08.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-433791696898 1-395136991238 2-356482285577' 2024-09-18T15:38:08.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:38:08.840 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-433791696898 2024-09-18T15:38:08.840 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:38:08.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:38:08.843 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-433791696898 2024-09-18T15:38:08.844 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:38:08.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=433791696898 2024-09-18T15:38:08.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 433791696898' 2024-09-18T15:38:08.846 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 433791696898 2024-09-18T15:38:08.847 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:38:09.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 433791696898 2024-09-18T15:38:09.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:38:10.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:38:10.222 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:38:10.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 433791696898 -lt 433791696898 2024-09-18T15:38:10.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:38:10.544 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-395136991238 2024-09-18T15:38:10.544 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:38:10.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:38:10.547 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-395136991238 2024-09-18T15:38:10.547 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:38:10.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=395136991238 2024-09-18T15:38:10.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 395136991238' 2024-09-18T15:38:10.550 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 395136991238 2024-09-18T15:38:10.551 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:38:10.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 395136991238 -lt 395136991238 2024-09-18T15:38:10.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:38:10.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-356482285577 2024-09-18T15:38:10.866 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:38:10.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:38:10.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-356482285577 2024-09-18T15:38:10.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:38:10.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=356482285577 2024-09-18T15:38:10.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 356482285577' 2024-09-18T15:38:10.871 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 356482285577 2024-09-18T15:38:10.872 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:38:11.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 356482285578 -lt 356482285577 2024-09-18T15:38:11.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:38:11.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:38:11.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:38:11.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:38:11.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:38:11.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:38:11.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:38:11.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:38:11.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:38:11.675 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:38:11.675 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:38:12.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:38:12.020 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:38:12.020 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:38:12.020 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:38:12.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:38:12.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:38:12.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:38:12.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3679: corrupt_scrub_erasure: echo -n bad-val 2024-09-18T15:38:12.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-18T15:38:12.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:38:12.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:38:12.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:38:12.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:38:12.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-18T15:38:12.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:38:12.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:38:12.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:38:12.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:38:12.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:38:12.390 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:38:12.390 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:38:12.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:38:12.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:38:12.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:38:12.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:38:12.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-18T15:38:12.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:38:12.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:38:12.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:38:12.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:38:12.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:38:12.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-18T15:38:15.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:38:15.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:38:15.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:38:15.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:38:15.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:38:15.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:38:15.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:38:15.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:38:15.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:38:15.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:38:15.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:38:15.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:38:15.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:38:15.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:38:15.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:38:15.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:38:15.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:38:15.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:38:15.273 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:38:15.273 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:38:15.273 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:38:15.274 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:38:15.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:38:15.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:38:15.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:38:15.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:38:15.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:38:15.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:38:15.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:38:15.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:38:15.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:38:15.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:38:15.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:38:15.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:38:15.277 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:38:15.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:38:15.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:38:15.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:38:15.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:38:15.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:38:15.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:38:15.311 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:38:15.302+0000 7f900f5a0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:38:15.314 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:38:15.306+0000 7f900f5a0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:38:15.317 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:38:15.310+0000 7f900f5a0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:38:15.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:38:15.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:38:15.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:38:15.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:38:15.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:38:15.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:15.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:38:15.598 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:38:15.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:15.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:38:15.858 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:38:15.850+0000 7f900f5a0540 -1 Falling back to public interface 2024-09-18T15:38:15.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:16.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:16.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:16.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:38:16.949 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:38:16.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:16.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:38:17.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:18.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:18.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:18.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:38:18.483 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:38:18.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:18.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:38:18.654 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:38:18.646+0000 7f900f5a0540 -1 osd.1 103 log_to_monitors true 2024-09-18T15:38:18.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:19.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:19.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:19.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:38:19.802 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:38:19.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:19.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:38:20.267 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 108 up_thru 108 down_at 104 last_clean_interval [92,103) [v2:127.0.0.1:6810/352764,v1:127.0.0.1:6811/352764] [v2:127.0.0.1:6812/352764,v1:127.0.0.1:6813/352764] exists,up 144c133f-b1d4-4eed-ab4d-dcfcf7bf1077 2024-09-18T15:38:20.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:38:20.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:38:20.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:38:20.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:38:20.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:38:20.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:38:20.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:38:20.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:38:20.269 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:38:20.269 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:38:20.269 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:38:20.269 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:38:20.269 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:38:20.269 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:38:20.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:38:20.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:38:20.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:38:20.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:38:20.485 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:38:20.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:38:20.852 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:38:20.852 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:38:20.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:38:20.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:38:20.853 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:38:20.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=433791696901 2024-09-18T15:38:20.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 433791696901 2024-09-18T15:38:20.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-433791696901' 2024-09-18T15:38:20.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:38:20.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:38:21.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=463856467970 2024-09-18T15:38:21.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 463856467970 2024-09-18T15:38:21.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-433791696901 1-463856467970' 2024-09-18T15:38:21.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:38:21.143 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:38:21.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=356482285581 2024-09-18T15:38:21.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 356482285581 2024-09-18T15:38:21.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-433791696901 1-463856467970 2-356482285581' 2024-09-18T15:38:21.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:38:21.285 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-433791696901 2024-09-18T15:38:21.285 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:38:21.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:38:21.288 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-433791696901 2024-09-18T15:38:21.289 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:38:21.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=433791696901 2024-09-18T15:38:21.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 433791696901' 2024-09-18T15:38:21.291 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 433791696901 2024-09-18T15:38:21.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:38:21.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 433791696900 -lt 433791696901 2024-09-18T15:38:21.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:38:22.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:38:22.599 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:38:22.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 433791696902 -lt 433791696901 2024-09-18T15:38:22.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:38:22.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-463856467970 2024-09-18T15:38:22.912 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:38:22.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:38:22.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-463856467970 2024-09-18T15:38:22.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:38:22.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=463856467970 2024-09-18T15:38:22.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 463856467970' 2024-09-18T15:38:22.918 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 463856467970 2024-09-18T15:38:22.919 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:38:23.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 463856467970 -lt 463856467970 2024-09-18T15:38:23.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:38:23.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-356482285581 2024-09-18T15:38:23.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:38:23.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:38:23.266 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-356482285581 2024-09-18T15:38:23.266 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:38:23.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=356482285581 2024-09-18T15:38:23.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 356482285581' 2024-09-18T15:38:23.268 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 356482285581 2024-09-18T15:38:23.269 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:38:23.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 356482285581 -lt 356482285581 2024-09-18T15:38:23.578 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:38:23.579 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:38:23.579 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:38:23.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:38:23.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:38:23.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:38:23.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:38:23.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:38:23.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:38:23.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:38:23.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:38:24.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:38:24.336 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:38:24.336 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:38:24.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:38:24.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:38:24.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:38:24.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:38:24.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3633: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-18T15:38:24.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objname=EOBJ7 2024-09-18T15:38:24.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3635: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-09-18T15:38:24.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:38:24.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2024-09-18T15:38:24.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=EOBJ7 2024-09-18T15:38:24.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:38:24.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:38:24.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:38:25.183 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:38:25.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:38:25.666 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:38:25.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:38:25.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:38:25.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:38:25.890 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: expr 7 % 2 2024-09-18T15:38:25.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3637: corrupt_scrub_erasure: local osd=1 2024-09-18T15:38:25.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: case $i in 2024-09-18T15:38:25.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-09-18T15:38:25.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3685: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-09-18T15:38:25.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-09-18T15:38:26.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3689: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-18T15:38:26.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:38:26.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:38:26.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:38:26.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:38:26.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-18T15:38:26.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:38:26.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:38:26.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:38:26.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:38:26.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:38:26.135 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:38:26.135 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:38:26.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:38:26.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:38:26.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:38:26.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:38:26.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-18T15:38:26.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:38:26.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:38:26.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:38:26.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:38:26.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:38:26.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-09-18T15:38:28.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-18T15:38:28.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:38:28.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:38:28.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:38:28.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:38:28.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:38:28.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:38:28.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:38:28.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:38:28.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:38:28.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:38:28.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:38:28.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:38:28.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:38:28.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:38:28.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:38:28.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:38:28.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:38:28.857 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:38:28.857 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:38:28.857 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:38:28.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:38:28.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:38:28.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:38:28.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:38:28.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:38:28.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:38:28.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:38:28.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:38:28.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:38:28.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:38:28.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:38:28.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:38:28.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:38:28.860 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:38:28.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:38:28.861 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:38:28.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:38:28.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:38:28.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:38:28.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:38:28.894 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:38:28.886+0000 7f03f6524540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:38:28.897 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:38:28.890+0000 7f03f6524540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:38:28.904 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:38:28.898+0000 7f03f6524540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:38:29.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:38:29.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:38:29.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:38:29.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:38:29.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:38:29.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:29.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:38:29.200 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:38:29.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:29.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:29.454 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:38:29.446+0000 7f03f6524540 -1 Falling back to public interface 2024-09-18T15:38:29.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:30.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:30.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:30.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:38:30.553 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:38:30.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:30.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:30.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:31.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:31.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:31.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:38:31.875 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:38:31.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:31.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:32.130 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:38:32.122+0000 7f03f6524540 -1 osd.0 113 log_to_monitors true 2024-09-18T15:38:32.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:33.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:33.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:33.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:38:33.192 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:38:33.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:33.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:33.544 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 118 up_thru 118 down_at 114 last_clean_interval [101,113) [v2:127.0.0.1:6802/354205,v1:127.0.0.1:6803/354205] [v2:127.0.0.1:6804/354205,v1:127.0.0.1:6805/354205] exists,up d366d8b9-5243-44c9-8553-5749a6e3a9c7 2024-09-18T15:38:33.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:38:33.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:38:33.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:38:33.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:38:33.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:38:33.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:38:33.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:38:33.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:38:33.546 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:38:33.546 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:38:33.546 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:38:33.547 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:38:33.547 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:38:33.547 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:38:33.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:38:33.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:38:33.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:38:33.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:38:33.784 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:38:34.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:38:34.233 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:38:34.234 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:38:34.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:38:34.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:38:34.234 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:38:34.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=506806140930 2024-09-18T15:38:34.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 506806140930 2024-09-18T15:38:34.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-506806140930' 2024-09-18T15:38:34.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:38:34.370 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:38:34.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=463856467974 2024-09-18T15:38:34.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 463856467974 2024-09-18T15:38:34.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-506806140930 1-463856467974' 2024-09-18T15:38:34.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:38:34.526 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:38:34.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=356482285585 2024-09-18T15:38:34.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 356482285585 2024-09-18T15:38:34.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-506806140930 1-463856467974 2-356482285585' 2024-09-18T15:38:34.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:38:34.678 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-506806140930 2024-09-18T15:38:34.678 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:38:34.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:38:34.682 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-506806140930 2024-09-18T15:38:34.682 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:38:34.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=506806140930 2024-09-18T15:38:34.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 506806140930' 2024-09-18T15:38:34.683 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 506806140930 2024-09-18T15:38:34.684 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:38:34.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 506806140930 -lt 506806140930 2024-09-18T15:38:34.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:38:34.987 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-463856467974 2024-09-18T15:38:34.987 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:38:34.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:38:34.991 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-463856467974 2024-09-18T15:38:34.991 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:38:34.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=463856467974 2024-09-18T15:38:34.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 463856467974' 2024-09-18T15:38:34.993 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 463856467974 2024-09-18T15:38:34.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:38:35.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 463856467973 -lt 463856467974 2024-09-18T15:38:35.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:38:36.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:38:36.339 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:38:36.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 463856467974 -lt 463856467974 2024-09-18T15:38:36.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:38:36.652 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-356482285585 2024-09-18T15:38:36.652 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:38:36.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:38:36.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-356482285585 2024-09-18T15:38:36.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:38:36.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=356482285585 2024-09-18T15:38:36.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 356482285585' 2024-09-18T15:38:36.659 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 356482285585 2024-09-18T15:38:36.659 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:38:37.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 356482285585 -lt 356482285585 2024-09-18T15:38:37.001 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:38:37.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:38:37.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:38:37.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:38:37.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:38:37.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:38:37.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:38:37.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:38:37.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:38:37.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:38:37.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:38:37.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:38:37.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:38:37.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:38:37.753 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:38:38.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:38:38.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:38:38.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:38:38.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-18T15:38:38.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:38:38.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:38:38.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:38:38.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:38:38.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-18T15:38:38.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:38:38.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:38:38.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:38:38.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:38:38.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:38:38.125 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:38:38.125 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:38:38.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:38:38.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:38:38.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:38:38.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:38:38.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-18T15:38:38.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:38:38.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:38:38.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:38:38.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:38:38.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:38:38.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-18T15:38:40.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-18T15:38:40.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:38:40.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:38:40.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:38:40.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:38:40.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:38:40.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:38:40.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:38:40.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:38:40.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:38:40.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:38:40.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:38:40.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:38:40.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:38:40.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:38:40.897 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:38:40.897 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:38:40.897 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:38:40.899 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:38:40.899 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:38:40.899 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:38:40.899 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:38:40.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:38:40.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:38:40.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:38:40.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:38:40.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:38:40.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:38:40.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:38:40.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:38:40.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:38:40.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:38:40.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:38:40.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:38:40.901 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:38:40.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:38:40.902 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:38:40.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:38:40.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:38:40.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:38:40.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:38:40.933 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:38:40.926+0000 7f101101d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:38:40.936 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:38:40.930+0000 7f101101d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:38:40.940 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:38:40.930+0000 7f101101d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:38:41.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:38:41.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:38:41.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:38:41.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:38:41.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:38:41.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:41.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:38:41.266 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:38:41.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:41.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:41.502 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:38:41.494+0000 7f101101d540 -1 Falling back to public interface 2024-09-18T15:38:41.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:42.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:42.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:42.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:38:42.618 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:38:42.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:42.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:42.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:43.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:43.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:43.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:38:43.929 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:38:43.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:43.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:44.027 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:38:44.018+0000 7f101101d540 -1 osd.0 120 log_to_monitors true 2024-09-18T15:38:44.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:45.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:45.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:45.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:38:45.341 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:38:45.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:45.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:45.774 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 125 up_thru 125 down_at 121 last_clean_interval [118,120) [v2:127.0.0.1:6802/355559,v1:127.0.0.1:6803/355559] [v2:127.0.0.1:6804/355559,v1:127.0.0.1:6805/355559] exists,up d366d8b9-5243-44c9-8553-5749a6e3a9c7 2024-09-18T15:38:45.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:38:45.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:38:45.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:38:45.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:38:45.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:38:45.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:38:45.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:38:45.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:38:45.775 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:38:45.776 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:38:45.776 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:38:45.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:38:45.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:38:45.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:38:46.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:38:46.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:38:46.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:38:46.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:38:46.008 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:38:46.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:38:46.319 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:38:46.319 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:38:46.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:38:46.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:38:46.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:38:46.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=536870912002 2024-09-18T15:38:46.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 536870912002 2024-09-18T15:38:46.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-536870912002' 2024-09-18T15:38:46.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:38:46.479 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:38:46.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=463856467977 2024-09-18T15:38:46.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 463856467977 2024-09-18T15:38:46.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-536870912002 1-463856467977' 2024-09-18T15:38:46.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:38:46.638 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:38:46.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=356482285588 2024-09-18T15:38:46.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 356482285588 2024-09-18T15:38:46.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-536870912002 1-463856467977 2-356482285588' 2024-09-18T15:38:46.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:38:46.782 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-536870912002 2024-09-18T15:38:46.782 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:38:46.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:38:46.785 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-536870912002 2024-09-18T15:38:46.785 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:38:46.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=536870912002 2024-09-18T15:38:46.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 536870912002' 2024-09-18T15:38:46.787 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 536870912002 2024-09-18T15:38:46.788 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:38:47.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 536870912002 2024-09-18T15:38:47.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:38:48.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:38:48.107 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:38:48.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 536870912002 2024-09-18T15:38:48.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:38:49.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-18T15:38:49.419 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:38:49.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 536870912002 -lt 536870912002 2024-09-18T15:38:49.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:38:49.739 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-463856467977 2024-09-18T15:38:49.739 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:38:49.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:38:49.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-463856467977 2024-09-18T15:38:49.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:38:49.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=463856467977 2024-09-18T15:38:49.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 463856467977' 2024-09-18T15:38:49.745 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 463856467977 2024-09-18T15:38:49.745 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:38:50.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 463856467977 -lt 463856467977 2024-09-18T15:38:50.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:38:50.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:38:50.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-356482285588 2024-09-18T15:38:50.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:38:50.076 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-356482285588 2024-09-18T15:38:50.076 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:38:50.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=356482285588 2024-09-18T15:38:50.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 356482285588' 2024-09-18T15:38:50.077 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 356482285588 2024-09-18T15:38:50.078 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:38:50.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 356482285588 -lt 356482285588 2024-09-18T15:38:50.418 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:38:50.419 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:38:50.419 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:38:50.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:38:50.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:38:50.847 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:38:50.847 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:38:50.847 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:38:50.848 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:38:50.848 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:38:50.848 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:38:51.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:38:51.195 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:38:51.196 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:38:51.196 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:38:51.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:38:51.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:38:51.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:38:51.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3691: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-09-18T15:38:51.561 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3697: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-09-18T15:38:51.561 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-09-18T15:38:51.561 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=EOBJ0 2024-09-18T15:38:51.561 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool EOBJ0 2024-09-18T15:38:51.561 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:38:51.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3697: corrupt_scrub_erasure: local pg=3.0 2024-09-18T15:38:51.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3699: corrupt_scrub_erasure: pg_scrub 3.0 2024-09-18T15:38:51.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: pg_scrub: local pgid=3.0 2024-09-18T15:38:51.900 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: pg_scrub: get_last_scrub_stamp 3.0 2024-09-18T15:38:51.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:38:51.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:38:51.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:38:51.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:38:52.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: pg_scrub: local last_scrub=2024-09-18T15:36:12.506316+0000 2024-09-18T15:38:52.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_scrub: ceph pg scrub 3.0 2024-09-18T15:38:52.539 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-09-18T15:38:52.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_scrub: wait_for_scrub 3.0 2024-09-18T15:36:12.506316+0000 2024-09-18T15:38:52.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=3.0 2024-09-18T15:38:52.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:36:12.506316+0000 2024-09-18T15:38:52.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:38:52.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:38:52.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:38:52.549 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:38:52.549 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:38:52.549 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:38:52.550 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:38:52.550 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:38:52.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:36:12.506316+0000 '>' 2024-09-18T15:36:12.506316+0000 2024-09-18T15:38:52.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:38:53.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:38:53.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:38:53.858 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:38:53.858 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:38:53.858 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:38:53.859 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:38:53.859 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:38:54.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:36:12.506316+0000 '>' 2024-09-18T15:36:12.506316+0000 2024-09-18T15:38:54.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:38:55.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:38:55.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:38:55.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:38:55.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:38:55.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:38:55.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:38:55.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:38:55.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:38:53.503916+0000 '>' 2024-09-18T15:36:12.506316+0000 2024-09-18T15:38:55.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:38:55.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3701: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-09-18T15:38:55.517 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3703: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-09-18T15:38:55.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3703: corrupt_scrub_erasure: test 1 = 1 2024-09-18T15:38:55.560 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3705: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-18T15:38:55.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3705: corrupt_scrub_erasure: test 3.0 = 3.0 2024-09-18T15:38:55.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3707: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-09-18T15:38:55.640 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3709: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-09-18T15:38:55.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3709: corrupt_scrub_erasure: epoch=125 2024-09-18T15:38:55.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3711: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-18T15:38:55.680 INFO:tasks.workunit.client.0.smithi046.stderr: . as $in 2024-09-18T15:38:55.680 INFO:tasks.workunit.client.0.smithi046.stderr: | if type == "object" then 2024-09-18T15:38:55.681 INFO:tasks.workunit.client.0.smithi046.stderr: reduce keys[] as $key 2024-09-18T15:38:55.681 INFO:tasks.workunit.client.0.smithi046.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-18T15:38:55.681 INFO:tasks.workunit.client.0.smithi046.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-18T15:38:55.681 INFO:tasks.workunit.client.0.smithi046.stderr: else f 2024-09-18T15:38:55.681 INFO:tasks.workunit.client.0.smithi046.stderr: end; 2024-09-18T15:38:55.681 INFO:tasks.workunit.client.0.smithi046.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-18T15:38:55.681 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-18T15:38:55.681 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-18T15:38:55.681 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-18T15:38:55.681 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-18T15:38:55.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3711: corrupt_scrub_erasure: jq .inconsistents 2024-09-18T15:38:55.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3711: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-18T15:38:55.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4331: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-18T15:38:55.736 INFO:tasks.workunit.client.0.smithi046.stderr: . as $in 2024-09-18T15:38:55.736 INFO:tasks.workunit.client.0.smithi046.stderr: | if type == "object" then 2024-09-18T15:38:55.736 INFO:tasks.workunit.client.0.smithi046.stderr: reduce keys[] as $key 2024-09-18T15:38:55.736 INFO:tasks.workunit.client.0.smithi046.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-18T15:38:55.736 INFO:tasks.workunit.client.0.smithi046.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-18T15:38:55.736 INFO:tasks.workunit.client.0.smithi046.stderr: else f 2024-09-18T15:38:55.737 INFO:tasks.workunit.client.0.smithi046.stderr: end; 2024-09-18T15:38:55.737 INFO:tasks.workunit.client.0.smithi046.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-18T15:38:55.737 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-18T15:38:55.737 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-18T15:38:55.737 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-18T15:38:55.737 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-18T15:38:55.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4331: corrupt_scrub_erasure: jq .inconsistents 2024-09-18T15:38:55.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4331: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-18T15:38:55.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-18T15:38:55.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-18T15:38:55.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4333: corrupt_scrub_erasure: test no = yes 2024-09-18T15:38:55.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4338: corrupt_scrub_erasure: test '' = yes 2024-09-18T15:38:55.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4343: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-09-18T15:38:55.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: pg_deep_scrub: local pgid=3.0 2024-09-18T15:38:55.790 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-18T15:38:55.790 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:38:55.790 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:38:55.791 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:38:55.791 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-18T15:38:56.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: pg_deep_scrub: local last_scrub=2024-09-18T15:36:12.506316+0000 2024-09-18T15:38:56.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-09-18T15:38:56.385 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-09-18T15:38:56.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: pg_deep_scrub: wait_for_scrub 3.0 2024-09-18T15:36:12.506316+0000 last_deep_scrub_stamp 2024-09-18T15:38:56.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=3.0 2024-09-18T15:38:56.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:36:12.506316+0000 2024-09-18T15:38:56.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-09-18T15:38:56.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:38:56.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:38:56.396 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-18T15:38:56.397 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:38:56.397 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:38:56.397 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:38:56.397 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-18T15:38:56.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:36:12.506316+0000 '>' 2024-09-18T15:36:12.506316+0000 2024-09-18T15:38:56.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:38:57.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:38:57.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:38:57.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-18T15:38:57.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:38:57.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:38:57.707 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:38:57.707 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-18T15:38:58.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:36:12.506316+0000 '>' 2024-09-18T15:36:12.506316+0000 2024-09-18T15:38:58.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:38:59.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:38:59.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:38:59.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-18T15:38:59.027 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:38:59.027 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:38:59.027 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:38:59.027 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-18T15:38:59.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:36:12.506316+0000 '>' 2024-09-18T15:36:12.506316+0000 2024-09-18T15:38:59.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:39:00.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:39:00.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:39:00.343 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-18T15:39:00.344 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:39:00.344 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:39:00.344 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:39:00.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-18T15:39:00.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:38:58.659632+0000 '>' 2024-09-18T15:36:12.506316+0000 2024-09-18T15:39:00.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:39:00.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4345: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-09-18T15:39:00.699 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4347: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-09-18T15:39:00.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4347: corrupt_scrub_erasure: test 1 = 1 2024-09-18T15:39:00.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4349: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-18T15:39:00.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4349: corrupt_scrub_erasure: test 3.0 = 3.0 2024-09-18T15:39:00.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4351: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-09-18T15:39:00.826 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4353: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-09-18T15:39:00.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4353: corrupt_scrub_erasure: epoch=125 2024-09-18T15:39:00.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4355: corrupt_scrub_erasure: '[' true = true ']' 2024-09-18T15:39:00.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4357: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-18T15:39:00.869 INFO:tasks.workunit.client.0.smithi046.stderr: . as $in 2024-09-18T15:39:00.869 INFO:tasks.workunit.client.0.smithi046.stderr: | if type == "object" then 2024-09-18T15:39:00.869 INFO:tasks.workunit.client.0.smithi046.stderr: reduce keys[] as $key 2024-09-18T15:39:00.869 INFO:tasks.workunit.client.0.smithi046.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-18T15:39:00.869 INFO:tasks.workunit.client.0.smithi046.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-18T15:39:00.869 INFO:tasks.workunit.client.0.smithi046.stderr: else f 2024-09-18T15:39:00.869 INFO:tasks.workunit.client.0.smithi046.stderr: end; 2024-09-18T15:39:00.869 INFO:tasks.workunit.client.0.smithi046.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-18T15:39:00.869 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-18T15:39:00.869 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-18T15:39:00.869 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-18T15:39:00.870 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-18T15:39:00.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4357: corrupt_scrub_erasure: jq .inconsistents 2024-09-18T15:39:00.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4357: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-18T15:39:00.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5745: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-18T15:39:00.928 INFO:tasks.workunit.client.0.smithi046.stderr: . as $in 2024-09-18T15:39:00.928 INFO:tasks.workunit.client.0.smithi046.stderr: | if type == "object" then 2024-09-18T15:39:00.928 INFO:tasks.workunit.client.0.smithi046.stderr: reduce keys[] as $key 2024-09-18T15:39:00.928 INFO:tasks.workunit.client.0.smithi046.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-18T15:39:00.928 INFO:tasks.workunit.client.0.smithi046.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-18T15:39:00.928 INFO:tasks.workunit.client.0.smithi046.stderr: else f 2024-09-18T15:39:00.929 INFO:tasks.workunit.client.0.smithi046.stderr: end; 2024-09-18T15:39:00.929 INFO:tasks.workunit.client.0.smithi046.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-18T15:39:00.929 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-18T15:39:00.929 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-18T15:39:00.929 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-18T15:39:00.929 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-18T15:39:00.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5745: corrupt_scrub_erasure: jq .inconsistents 2024-09-18T15:39:00.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5745: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-18T15:39:00.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5746: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-18T15:39:00.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-18T15:39:00.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5747: corrupt_scrub_erasure: test no = yes 2024-09-18T15:39:00.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5758: corrupt_scrub_erasure: test '' = yes 2024-09-18T15:39:00.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5763: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-09-18T15:39:01.480 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'ecpool' removed 2024-09-18T15:39:01.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T15:39:01.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:39:01.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:39:01.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:39:01.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:39:01.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:39:01.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:39:01.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:39:01.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:39:01.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:39:01.819 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:39:01.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:39:01.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:39:01.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:39:01.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:39:01.825 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:39:01.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:39:01.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:39:01.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:39:01.828 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:39:01.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:39:01.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:39:01.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:39:01.866 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:39:01.866 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:39:01.866 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:39:01.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:39:01.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:39:01.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:39:01.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T15:39:01.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T15:39:01.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T15:39:01.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T15:39:01.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:39:01.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:39:01.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:39:01.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:39:01.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:39:01.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:39:01.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:39:01.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:39:01.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:39:01.876 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:39:01.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:39:01.878 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:39:01.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:39:01.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:39:01.881 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:39:01.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:39:01.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:39:01.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:39:01.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:39:01.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:39:01.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:39:01.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:39:01.892 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:39:01.892 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:39:01.892 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:39:01.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:39:01.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:39:01.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:39:01.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T15:39:01.897 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:39:01.897 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:39:01.897 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:39:01.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T15:39:01.900 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:39:01.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:39:01.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:39:01.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T15:39:01.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_corrupt_scrub_replicated td/osd-scrub-repair 2024-09-18T15:39:01.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: local dir=td/osd-scrub-repair 2024-09-18T15:39:01.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1009: TEST_corrupt_scrub_replicated: local poolname=csr_pool 2024-09-18T15:39:01.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: local total_objs=19 2024-09-18T15:39:01.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1012: TEST_corrupt_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-09-18T15:39:01.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T15:39:01.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:39:01.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:39:01.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:39:01.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T15:39:01.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-09-18T15:39:02.290 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:39:02.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:39:02.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:39:02.291 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:39:02.291 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:39:02.291 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:39:02.292 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:39:02.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-18T15:39:02.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:39:02.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:39:02.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:39:02.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:39:02.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:39:02.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:39:02.583 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:39:02.583 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:39:02.583 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:39:02.583 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:39:02.583 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:39:02.583 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:39:02.584 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:39:02.584 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:39:02.584 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T15:39:02.675 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:39:02.675 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:39:02.675 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:39:02.675 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:39:02.676 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:39:02.676 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:39:02.676 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:39:02.676 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:39:02.677 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:39:02.677 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:39:02.677 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:39:02.677 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:39:02.677 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:39:02.677 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T15:39:02.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1013: TEST_corrupt_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-09-18T15:39:02.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T15:39:02.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:39:02.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:39:02.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:39:02.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T15:39:02.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:39:03.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:39:03.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:39:03.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:39:03.630 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:39:03.631 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:39:03.631 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:39:03.631 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:39:03.632 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:39:03.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:39:03.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1014: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 0 2024-09-18T15:39:03.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:39:03.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:39:03.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:39:03.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:39:03.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:39:03.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:39:03.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:39:03.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:39:03.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:39:03.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:39:03.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:39:03.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:39:03.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:39:03.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:39:03.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:39:03.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:39:03.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:39:03.674 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:39:03.674 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:39:03.675 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:39:03.675 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:39:03.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:39:03.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:39:03.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:39:03.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:39:03.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:39:03.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:39:03.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:39:03.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:39:03.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:39:03.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:39:03.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:39:03.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:39:03.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:39:03.676 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:39:03.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=537f9958-5809-4e44-8060-20ee457d6502 2024-09-18T15:39:03.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 537f9958-5809-4e44-8060-20ee457d6502' 2024-09-18T15:39:03.677 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 537f9958-5809-4e44-8060-20ee457d6502 2024-09-18T15:39:03.677 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:39:03.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAX9Opm4hoYKRAAY9CxnPqs6i1RBl8yn4E3Pg== 2024-09-18T15:39:03.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAX9Opm4hoYKRAAY9CxnPqs6i1RBl8yn4E3Pg=="}' 2024-09-18T15:39:03.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 537f9958-5809-4e44-8060-20ee457d6502 -i td/osd-scrub-repair/0/new.json 2024-09-18T15:39:03.980 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:39:03.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-18T15:39:03.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAX9Opm4hoYKRAAY9CxnPqs6i1RBl8yn4E3Pg== --osd-uuid 537f9958-5809-4e44-8060-20ee457d6502 2024-09-18T15:39:04.026 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:39:04.018+0000 7f112aaa7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:39:04.028 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:39:04.022+0000 7f112aaa7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:39:04.030 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:39:04.022+0000 7f112aaa7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:39:04.031 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:39:04.022+0000 7f112aaa7540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:39:04.031 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:39:04.022+0000 7f112aaa7540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:39:04.031 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:39:04.022+0000 7f112aaa7540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:39:04.115 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:39:04.106+0000 7f112aaa7540 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-18T15:39:09.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T15:39:09.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:39:09.687 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T15:39:09.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:39:09.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:39:10.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:39:10.072 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T15:39:10.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:39:10.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:39:10.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:39:10.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:39:10.102 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:39:10.094+0000 7f7f1c7f4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:39:10.106 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:39:10.098+0000 7f7f1c7f4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:39:10.109 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:39:10.098+0000 7f7f1c7f4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:39:10.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:39:10.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:39:10.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:39:10.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:39:10.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:39:10.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:10.414 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:39:10.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:39:10.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:10.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:39:10.666 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:39:10.658+0000 7f7f1c7f4540 -1 Falling back to public interface 2024-09-18T15:39:10.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:11.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:11.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:11.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:39:11.766 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:39:11.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:11.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:39:12.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:13.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:13.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:13.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:39:13.081 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:39:13.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:13.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:39:13.375 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:39:13.366+0000 7f7f1c7f4540 -1 osd.0 0 log_to_monitors true 2024-09-18T15:39:13.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:14.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:14.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:14.396 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:39:14.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:39:14.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:14.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:39:14.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:15.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:15.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:15.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:39:15.711 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:39:15.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:15.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:39:16.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:17.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:17.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:17.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:39:17.097 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:39:17.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:17.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:39:17.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:18.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:18.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:18.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:39:18.414 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:39:18.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:18.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:39:18.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:19.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:19.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:19.734 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:39:19.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:39:19.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:19.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:39:20.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:21.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:21.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:21.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:39:21.098 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:39:21.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:21.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:39:21.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:22.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:22.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:22.411 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:39:22.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:39:22.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:22.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:39:22.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:23.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:23.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:23.729 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:39:23.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:39:23.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:23.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:39:24.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:25.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:25.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:25.234 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:39:25.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:39:25.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:25.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:39:25.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:26.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:26.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:26.551 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:39:26.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:39:26.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:26.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:39:26.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:27.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:27.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:27.981 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:39:27.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:39:27.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:27.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:39:28.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:29.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:29.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:29.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:39:29.523 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:39:29.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:29.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:39:30.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:31.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:31.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:31.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:39:31.011 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:39:31.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:31.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:39:31.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:32.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:32.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:32.507 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:39:32.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:39:32.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:32.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:39:33.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:34.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:34.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:34.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:39:34.118 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:39:34.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:34.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:39:34.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:35.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:35.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:35.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:39:35.832 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:39:35.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:35.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:39:36.151 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/358100,v1:127.0.0.1:6803/358100] [v2:127.0.0.1:6804/358100,v1:127.0.0.1:6805/358100] exists,up 537f9958-5809-4e44-8060-20ee457d6502 2024-09-18T15:39:36.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:39:36.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:39:36.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:39:36.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1015: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 1 2024-09-18T15:39:36.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:39:36.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:39:36.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:39:36.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:39:36.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:39:36.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:39:36.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:39:36.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:39:36.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:39:36.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:39:36.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:39:36.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:39:36.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:39:36.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:39:36.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:39:36.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:39:36.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:39:36.157 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:39:36.157 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:39:36.157 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:39:36.157 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:39:36.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:39:36.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:39:36.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:39:36.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:39:36.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:39:36.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:39:36.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:39:36.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:39:36.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:39:36.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:39:36.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:39:36.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:39:36.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:39:36.159 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:39:36.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8c8956ed-6b0c-4a8b-8558-811b867c0d90 2024-09-18T15:39:36.160 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 8c8956ed-6b0c-4a8b-8558-811b867c0d90 2024-09-18T15:39:36.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 8c8956ed-6b0c-4a8b-8558-811b867c0d90' 2024-09-18T15:39:36.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:39:36.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA49OpmiSyJChAAkFPgPbbr2GcEsvXFHJyqdw== 2024-09-18T15:39:36.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA49OpmiSyJChAAkFPgPbbr2GcEsvXFHJyqdw=="}' 2024-09-18T15:39:36.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8c8956ed-6b0c-4a8b-8558-811b867c0d90 -i td/osd-scrub-repair/1/new.json 2024-09-18T15:39:36.656 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:39:36.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-18T15:39:36.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA49OpmiSyJChAAkFPgPbbr2GcEsvXFHJyqdw== --osd-uuid 8c8956ed-6b0c-4a8b-8558-811b867c0d90 2024-09-18T15:39:36.708 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:39:36.701+0000 7f753de6f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:39:36.712 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:39:36.705+0000 7f753de6f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:39:36.714 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:39:36.705+0000 7f753de6f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:39:36.715 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:39:36.705+0000 7f753de6f540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:39:36.715 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:39:36.705+0000 7f753de6f540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:39:36.715 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:39:36.705+0000 7f753de6f540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:39:36.782 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:39:36.773+0000 7f753de6f540 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-18T15:39:43.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-18T15:39:43.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:39:43.376 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T15:39:43.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:39:43.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:39:43.771 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T15:39:43.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:39:43.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:39:43.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:39:43.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:39:43.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:39:43.800 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:39:43.793+0000 7f119c2ac540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:39:43.828 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:39:43.817+0000 7f119c2ac540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:39:43.829 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:39:43.821+0000 7f119c2ac540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:39:44.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:39:44.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:39:44.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:39:44.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:39:44.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:39:44.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:44.065 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:39:44.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:39:44.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:44.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:39:44.370 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:39:44.361+0000 7f119c2ac540 -1 Falling back to public interface 2024-09-18T15:39:44.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:45.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:45.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:45.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:39:45.421 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:39:45.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:45.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:39:45.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:46.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:46.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:46.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:39:46.733 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:39:46.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:46.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:39:47.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:47.168 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:39:47.149+0000 7f119c2ac540 -1 osd.1 0 log_to_monitors true 2024-09-18T15:39:48.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:48.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:48.090 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:39:48.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:39:48.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:48.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:39:48.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:49.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:49.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:49.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:39:49.458 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:39:49.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:49.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:39:49.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:50.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:50.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:50.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:39:50.809 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:39:50.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:50.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:39:51.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:52.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:52.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:52.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:39:52.235 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:39:52.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:52.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:39:52.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:53.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:53.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:53.549 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:39:53.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:39:53.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:53.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:39:53.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:54.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:54.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:54.869 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:39:54.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:39:54.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:54.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:39:55.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:56.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:56.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:56.248 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:39:56.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:39:56.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:56.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:39:56.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:57.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:57.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:57.566 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:39:57.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:39:57.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:57.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:39:58.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:59.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:59.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:59.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:39:59.012 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:39:59.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:59.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:39:59.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:40:00.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:40:00.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:00.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:40:00.372 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:40:00.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:00.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:40:00.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:40:01.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:40:01.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:01.780 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:40:01.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:40:01.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:01.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:40:02.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:40:03.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:40:03.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:03.258 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:40:03.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:40:03.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:03.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:40:03.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:40:04.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:40:04.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:04.816 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:40:04.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:40:04.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:04.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:40:05.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:40:06.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:40:06.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:06.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:40:06.494 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:40:06.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:06.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:40:06.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:40:07.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:40:07.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:07.978 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:40:07.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:40:07.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:07.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:40:08.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:40:09.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:40:09.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:09.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:40:09.292 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:40:09.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:09.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:40:09.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:40:10.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:40:10.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:10.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:40:10.652 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:40:10.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:10.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:40:10.967 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/359879,v1:127.0.0.1:6811/359879] [v2:127.0.0.1:6812/359879,v1:127.0.0.1:6813/359879] exists,up 8c8956ed-6b0c-4a8b-8558-811b867c0d90 2024-09-18T15:40:10.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:40:10.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:40:10.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:40:10.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: TEST_corrupt_scrub_replicated: create_rbd_pool 2024-09-18T15:40:10.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T15:40:11.353 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' does not exist 2024-09-18T15:40:11.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T15:40:11.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T15:40:11.814 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' created 2024-09-18T15:40:11.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:40:12.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T15:40:14.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: TEST_corrupt_scrub_replicated: wait_for_clean 2024-09-18T15:40:14.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:40:14.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:40:14.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:40:14.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:40:14.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:40:14.868 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:40:14.868 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:40:14.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:40:14.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:40:14.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:40:15.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:40:15.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:40:15.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:40:15.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:40:15.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:40:15.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:40:15.420 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:40:15.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:40:15.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:40:15.421 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:40:15.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836492 2024-09-18T15:40:15.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836492 2024-09-18T15:40:15.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836492' 2024-09-18T15:40:15.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:40:15.601 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:40:15.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672968 2024-09-18T15:40:15.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672968 2024-09-18T15:40:15.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836492 1-42949672968' 2024-09-18T15:40:15.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:40:15.745 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836492 2024-09-18T15:40:15.745 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:40:15.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:40:15.748 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836492 2024-09-18T15:40:15.748 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:40:15.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836492 2024-09-18T15:40:15.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-09-18T15:40:15.751 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836492 2024-09-18T15:40:15.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:40:16.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836491 -lt 21474836492 2024-09-18T15:40:16.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:40:17.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:40:17.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:40:17.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836493 -lt 21474836492 2024-09-18T15:40:17.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:40:17.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672968 2024-09-18T15:40:17.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:40:17.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:40:17.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672968 2024-09-18T15:40:17.364 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:40:17.366 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672968 2024-09-18T15:40:17.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672968 2024-09-18T15:40:17.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-09-18T15:40:17.366 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:40:17.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672968 -lt 42949672968 2024-09-18T15:40:17.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:40:17.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:40:17.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:40:18.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T15:40:18.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:40:18.116 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:40:18.117 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:40:18.117 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:40:18.117 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:40:18.117 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:40:18.117 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:40:18.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T15:40:18.429 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:40:18.429 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:40:18.430 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:40:18.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T15:40:18.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:40:18.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:40:18.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: create_pool foo 1 2024-09-18T15:40:18.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-09-18T15:40:19.287 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'foo' created 2024-09-18T15:40:19.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:40:20.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: create_pool csr_pool 1 1 2024-09-18T15:40:20.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2024-09-18T15:40:20.777 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'csr_pool' created 2024-09-18T15:40:20.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:40:21.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1021: TEST_corrupt_scrub_replicated: wait_for_clean 2024-09-18T15:40:21.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:40:21.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:40:21.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:40:21.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:40:21.793 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:40:21.794 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:40:21.794 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:40:21.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:40:21.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:40:21.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:40:22.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:40:22.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:40:22.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:40:22.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:40:22.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:40:22.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:40:22.413 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:40:22.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:40:22.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:40:22.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:40:22.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836495 2024-09-18T15:40:22.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836495 2024-09-18T15:40:22.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836495' 2024-09-18T15:40:22.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:40:22.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:40:22.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672970 2024-09-18T15:40:22.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672970 2024-09-18T15:40:22.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836495 1-42949672970' 2024-09-18T15:40:22.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:40:22.731 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836495 2024-09-18T15:40:22.732 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:40:22.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:40:22.735 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836495 2024-09-18T15:40:22.735 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:40:22.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836495 2024-09-18T15:40:22.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-09-18T15:40:22.737 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836495 2024-09-18T15:40:22.738 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:40:23.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836494 -lt 21474836495 2024-09-18T15:40:23.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:40:24.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:40:24.047 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:40:24.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836495 -lt 21474836495 2024-09-18T15:40:24.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:40:24.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672970 2024-09-18T15:40:24.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:40:24.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:40:24.364 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672970 2024-09-18T15:40:24.364 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:40:24.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672970 2024-09-18T15:40:24.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-09-18T15:40:24.367 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672970 2024-09-18T15:40:24.367 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:40:24.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672970 -lt 42949672970 2024-09-18T15:40:24.712 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:40:24.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:40:24.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:40:25.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:40:25.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:40:25.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:40:25.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:40:25.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:40:25.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:40:25.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:40:25.119 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:40:25.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:40:25.424 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:40:25.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:40:25.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:40:25.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:40:25.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:40:25.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:40:25.794 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: seq 1 19 2024-09-18T15:40:25.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:40:25.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-09-18T15:40:25.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-09-18T15:40:25.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:40:25.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2024-09-18T15:40:25.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ1 2024-09-18T15:40:25.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:40:25.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:40:25.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:40:26.283 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:40:26.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:40:26.801 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:40:26.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:40:26.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:40:26.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:40:26.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-09-18T15:40:27.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1028: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-09-18T15:40:27.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:40:27.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-09-18T15:40:27.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-09-18T15:40:27.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:40:27.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2024-09-18T15:40:27.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ2 2024-09-18T15:40:27.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:40:27.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:40:27.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:40:27.753 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:40:27.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:40:28.244 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:40:28.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:40:28.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:40:28.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:40:28.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-09-18T15:40:28.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1028: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-09-18T15:40:28.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:40:28.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-09-18T15:40:28.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ3 2024-09-18T15:40:28.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:40:28.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2024-09-18T15:40:28.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ3 2024-09-18T15:40:28.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:40:28.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:40:28.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:40:29.368 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:40:29.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:40:29.952 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:40:29.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:40:29.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:40:29.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ3 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:40:30.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ3 hdr-ROBJ3 2024-09-18T15:40:30.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1028: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ3 key-ROBJ3 val-ROBJ3 2024-09-18T15:40:30.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:40:30.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-09-18T15:40:30.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ4 2024-09-18T15:40:30.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:40:30.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2024-09-18T15:40:30.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ4 2024-09-18T15:40:30.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:40:30.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:40:30.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:40:30.928 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:40:30.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:40:31.439 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:40:31.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:40:31.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:40:31.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ4 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:40:31.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ4 hdr-ROBJ4 2024-09-18T15:40:31.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1028: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ4 key-ROBJ4 val-ROBJ4 2024-09-18T15:40:31.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:40:31.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-09-18T15:40:31.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ5 2024-09-18T15:40:31.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:40:31.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2024-09-18T15:40:31.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ5 2024-09-18T15:40:31.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:40:31.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:40:31.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:40:32.427 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:40:32.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:40:32.922 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:40:32.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:40:32.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:40:32.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ5 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:40:33.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ5 hdr-ROBJ5 2024-09-18T15:40:33.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1028: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ5 key-ROBJ5 val-ROBJ5 2024-09-18T15:40:33.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:40:33.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-09-18T15:40:33.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ6 2024-09-18T15:40:33.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:40:33.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2024-09-18T15:40:33.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ6 2024-09-18T15:40:33.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:40:33.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:40:33.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:40:34.098 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:40:34.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:40:34.632 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:40:34.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:40:34.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:40:34.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ6 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:40:34.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ6 hdr-ROBJ6 2024-09-18T15:40:35.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1028: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ6 key-ROBJ6 val-ROBJ6 2024-09-18T15:40:35.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:40:35.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-09-18T15:40:35.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ7 2024-09-18T15:40:35.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:40:35.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2024-09-18T15:40:35.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ7 2024-09-18T15:40:35.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:40:35.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:40:35.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:40:35.684 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:40:35.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:40:36.205 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:40:36.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:40:36.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:40:36.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ7 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:40:36.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ7 hdr-ROBJ7 2024-09-18T15:40:36.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1028: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ7 key-ROBJ7 val-ROBJ7 2024-09-18T15:40:36.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:40:36.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-09-18T15:40:36.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ8 2024-09-18T15:40:36.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:40:36.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2024-09-18T15:40:36.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ8 2024-09-18T15:40:36.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:40:36.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:40:36.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:40:37.116 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:40:37.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:40:37.599 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:40:37.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:40:37.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:40:37.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ8 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:40:37.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ8 hdr-ROBJ8 2024-09-18T15:40:38.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1028: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ8 key-ROBJ8 val-ROBJ8 2024-09-18T15:40:38.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:40:38.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-09-18T15:40:38.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ9 2024-09-18T15:40:38.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:40:38.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2024-09-18T15:40:38.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ9 2024-09-18T15:40:38.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:40:38.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:40:38.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:40:38.578 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:40:38.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:40:39.114 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:40:39.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:40:39.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:40:39.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:40:39.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ9 hdr-ROBJ9 2024-09-18T15:40:39.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1028: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ9 key-ROBJ9 val-ROBJ9 2024-09-18T15:40:39.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:40:39.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-09-18T15:40:39.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ10 2024-09-18T15:40:39.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:40:39.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2024-09-18T15:40:39.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ10 2024-09-18T15:40:39.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:40:39.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:40:39.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:40:40.109 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:40:40.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:40:40.594 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:40:40.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:40:40.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:40:40.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ10 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:40:40.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ10 hdr-ROBJ10 2024-09-18T15:40:40.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1028: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key-ROBJ10 val-ROBJ10 2024-09-18T15:40:41.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:40:41.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-09-18T15:40:41.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ11 2024-09-18T15:40:41.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:40:41.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2024-09-18T15:40:41.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ11 2024-09-18T15:40:41.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:40:41.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:40:41.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:40:41.612 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:40:41.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:40:42.165 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:40:42.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:40:42.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:40:42.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ11 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:40:42.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ11 hdr-ROBJ11 2024-09-18T15:40:42.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1028: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ11 key-ROBJ11 val-ROBJ11 2024-09-18T15:40:42.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:40:42.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-09-18T15:40:42.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ12 2024-09-18T15:40:42.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:40:42.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2024-09-18T15:40:42.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ12 2024-09-18T15:40:42.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:40:42.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:40:42.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:40:43.084 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:40:43.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:40:43.593 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:40:43.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:40:43.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:40:43.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ12 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:40:43.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ12 hdr-ROBJ12 2024-09-18T15:40:43.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1028: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ12 key-ROBJ12 val-ROBJ12 2024-09-18T15:40:44.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:40:44.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-09-18T15:40:44.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ13 2024-09-18T15:40:44.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:40:44.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2024-09-18T15:40:44.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ13 2024-09-18T15:40:44.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:40:44.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:40:44.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:40:44.554 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:40:44.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:40:45.041 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:40:45.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:40:45.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:40:45.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ13 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:40:45.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ13 hdr-ROBJ13 2024-09-18T15:40:45.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1028: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ13 key-ROBJ13 val-ROBJ13 2024-09-18T15:40:45.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:40:45.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-09-18T15:40:45.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ14 2024-09-18T15:40:45.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:40:45.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2024-09-18T15:40:45.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ14 2024-09-18T15:40:45.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:40:45.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:40:45.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:40:46.079 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:40:46.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:40:46.614 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:40:46.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:40:46.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:40:46.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ14 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:40:46.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ14 hdr-ROBJ14 2024-09-18T15:40:47.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1028: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ14 key-ROBJ14 val-ROBJ14 2024-09-18T15:40:47.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:40:47.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-09-18T15:40:47.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ15 2024-09-18T15:40:47.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:40:47.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2024-09-18T15:40:47.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ15 2024-09-18T15:40:47.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:40:47.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:40:47.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:40:47.609 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:40:47.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:40:48.101 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:40:48.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:40:48.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:40:48.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ15 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:40:48.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ15 hdr-ROBJ15 2024-09-18T15:40:48.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1028: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ15 key-ROBJ15 val-ROBJ15 2024-09-18T15:40:48.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:40:48.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-09-18T15:40:48.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ16 2024-09-18T15:40:48.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:40:48.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2024-09-18T15:40:48.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ16 2024-09-18T15:40:48.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:40:48.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:40:48.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:40:49.130 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:40:49.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:40:49.672 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:40:49.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:40:49.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:40:49.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ16 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:40:49.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ16 hdr-ROBJ16 2024-09-18T15:40:50.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1028: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ16 key-ROBJ16 val-ROBJ16 2024-09-18T15:40:50.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:40:50.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-09-18T15:40:50.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ17 2024-09-18T15:40:50.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:40:50.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2024-09-18T15:40:50.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ17 2024-09-18T15:40:50.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:40:50.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:40:50.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:40:50.626 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:40:50.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:40:51.196 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:40:51.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:40:51.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:40:51.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ17 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:40:51.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ17 hdr-ROBJ17 2024-09-18T15:40:51.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1028: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ17 key-ROBJ17 val-ROBJ17 2024-09-18T15:40:51.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:40:51.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-09-18T15:40:51.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ18 2024-09-18T15:40:51.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:40:51.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2024-09-18T15:40:51.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ18 2024-09-18T15:40:51.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:40:51.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:40:51.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:40:52.173 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:40:52.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:40:52.710 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:40:52.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:40:52.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:40:52.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ18 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:40:53.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ18 hdr-ROBJ18 2024-09-18T15:40:53.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1028: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ18 key-ROBJ18 val-ROBJ18 2024-09-18T15:40:53.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:40:53.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1024: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-09-18T15:40:53.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ19 2024-09-18T15:40:53.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:40:53.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2024-09-18T15:40:53.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ19 2024-09-18T15:40:53.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:40:53.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:40:53.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:40:53.763 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:40:53.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:40:54.313 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:40:54.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:40:54.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:40:54.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:40:54.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ19 hdr-ROBJ19 2024-09-18T15:40:54.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1028: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ19 key-ROBJ19 val-ROBJ19 2024-09-18T15:40:54.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1032: TEST_corrupt_scrub_replicated: dd if=/dev/zero of=td/osd-scrub-repair/new.ROBJ19 bs=1024 count=1025 2024-09-18T15:40:54.862 INFO:tasks.workunit.client.0.smithi046.stderr:1025+0 records in 2024-09-18T15:40:54.862 INFO:tasks.workunit.client.0.smithi046.stderr:1025+0 records out 2024-09-18T15:40:54.862 INFO:tasks.workunit.client.0.smithi046.stderr:1049600 bytes (1.0 MB, 1.0 MiB) copied, 0.00639124 s, 164 MB/s 2024-09-18T15:40:54.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/new.ROBJ19 2024-09-18T15:40:55.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1034: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ19 2024-09-18T15:40:55.042 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1036: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-09-18T15:40:55.042 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-09-18T15:40:55.042 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-09-18T15:40:55.042 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-09-18T15:40:55.043 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:40:55.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1036: TEST_corrupt_scrub_replicated: local pg=3.0 2024-09-18T15:40:55.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1037: TEST_corrupt_scrub_replicated: get_primary csr_pool ROBJ0 2024-09-18T15:40:55.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=csr_pool 2024-09-18T15:40:55.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=ROBJ0 2024-09-18T15:40:55.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-09-18T15:40:55.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:40:55.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1037: TEST_corrupt_scrub_replicated: local primary=1 2024-09-18T15:40:55.716 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1040: TEST_corrupt_scrub_replicated: get_asok_path osd.0 2024-09-18T15:40:55.716 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T15:40:55.717 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T15:40:55.717 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:40:55.717 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:40:55.718 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:40:55.718 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-osd.0.asok 2024-09-18T15:40:55.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1040: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-09-18T15:40:55.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1040: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.137209/ceph-osd.0.asok config set osd_deep_scrub_update_digest_min_age 0 2024-09-18T15:40:55.803 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T15:40:55.803 INFO:tasks.workunit.client.0.smithi046.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-09-18T15:40:55.803 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T15:40:55.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1042: TEST_corrupt_scrub_replicated: get_asok_path osd.1 2024-09-18T15:40:55.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:40:55.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:40:55.812 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:40:55.813 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:40:55.813 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:40:55.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-osd.1.asok 2024-09-18T15:40:55.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1042: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-09-18T15:40:55.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1042: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.137209/ceph-osd.1.asok config set osd_deep_scrub_update_digest_min_age 0 2024-09-18T15:40:55.901 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T15:40:55.901 INFO:tasks.workunit.client.0.smithi046.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-09-18T15:40:55.901 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T15:40:55.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1044: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-09-18T15:40:55.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: pg_deep_scrub: local pgid=3.0 2024-09-18T15:40:55.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-18T15:40:55.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:40:55.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:40:55.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:40:55.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-18T15:40:56.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: pg_deep_scrub: local last_scrub=2024-09-18T15:40:20.683901+0000 2024-09-18T15:40:56.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-09-18T15:40:56.512 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-09-18T15:40:56.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: pg_deep_scrub: wait_for_scrub 3.0 2024-09-18T15:40:20.683901+0000 last_deep_scrub_stamp 2024-09-18T15:40:56.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=3.0 2024-09-18T15:40:56.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:40:20.683901+0000 2024-09-18T15:40:56.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-09-18T15:40:56.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:40:56.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:40:56.522 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-18T15:40:56.522 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:40:56.522 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:40:56.522 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:40:56.523 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-18T15:40:56.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:40:20.683901+0000 '>' 2024-09-18T15:40:20.683901+0000 2024-09-18T15:40:56.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:40:57.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:40:57.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:40:57.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-18T15:40:57.832 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:40:57.832 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:40:57.832 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:40:57.832 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-18T15:40:58.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:40:20.683901+0000 '>' 2024-09-18T15:40:20.683901+0000 2024-09-18T15:40:58.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:40:59.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:40:59.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:40:59.146 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-18T15:40:59.146 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:40:59.146 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:40:59.147 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:40:59.147 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-18T15:40:59.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:40:20.683901+0000 '>' 2024-09-18T15:40:20.683901+0000 2024-09-18T15:40:59.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:41:00.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:41:00.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:41:00.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-18T15:41:00.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:41:00.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:41:00.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:41:00.462 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-18T15:41:00.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:40:20.683901+0000 '>' 2024-09-18T15:40:20.683901+0000 2024-09-18T15:41:00.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:41:01.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:41:01.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:41:01.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-18T15:41:01.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:41:01.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:41:01.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:41:01.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-18T15:41:02.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:40:57.142428+0000 '>' 2024-09-18T15:40:20.683901+0000 2024-09-18T15:41:02.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:41:02.087 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: seq 1 19 2024-09-18T15:41:02.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:41:02.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-09-18T15:41:02.088 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: expr 1 % 2 2024-09-18T15:41:02.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: local osd=1 2024-09-18T15:41:02.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: case $i in 2024-09-18T15:41:02.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1055: TEST_corrupt_scrub_replicated: local payload=UVWXYZZZ 2024-09-18T15:41:02.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1056: TEST_corrupt_scrub_replicated: echo UVWXYZZZ 2024-09-18T15:41:02.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1057: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:41:02.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:41:02.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:41:02.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:41:02.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:41:02.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:41:02.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:41:02.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:41:02.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:41:02.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:41:02.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:41:02.093 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:41:02.093 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:41:02.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:41:02.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:41:02.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:41:02.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:41:02.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:41:02.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:41:02.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:41:02.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:41:02.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:41:02.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:41:02.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:41:04.020 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 3.0_head,#-5:00000000:::scrub_3.0:head#, (61) No data available 2024-09-18T15:41:04.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:41:04.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:41:04.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:41:04.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:41:04.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:41:04.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:41:04.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:41:04.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:41:04.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:41:04.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:41:04.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:41:04.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:41:04.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:41:04.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:41:04.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:41:04.592 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:41:04.592 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:41:04.592 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:41:04.594 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:41:04.594 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:41:04.594 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:41:04.594 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:41:04.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:41:04.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:41:04.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:41:04.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:41:04.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:41:04.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:41:04.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:41:04.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:41:04.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:41:04.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:41:04.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:41:04.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:41:04.595 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:41:04.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:41:04.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:41:04.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:41:04.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:41:04.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:41:04.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:41:04.628 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:41:04.620+0000 7f9c193fb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:41:04.631 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:41:04.620+0000 7f9c193fb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:41:04.633 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:41:04.624+0000 7f9c193fb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:41:04.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:41:04.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:41:04.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:41:04.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:41:04.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:41:04.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:04.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:41:04.914 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:41:04.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:04.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:05.167 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:41:05.156+0000 7f9c193fb540 -1 Falling back to public interface 2024-09-18T15:41:05.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:06.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:06.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:06.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:41:06.294 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:41:06.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:06.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:06.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:07.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:07.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:07.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:41:07.612 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:41:07.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:07.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:07.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:07.957 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:41:07.948+0000 7f9c193fb540 -1 osd.1 64 log_to_monitors true 2024-09-18T15:41:08.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:08.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:08.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:41:08.959 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:41:08.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:08.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:09.321 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 68 up_thru 68 down_at 65 last_clean_interval [10,64) [v2:127.0.0.1:6810/364303,v1:127.0.0.1:6811/364303] [v2:127.0.0.1:6812/364303,v1:127.0.0.1:6813/364303] exists,up 8c8956ed-6b0c-4a8b-8558-811b867c0d90 2024-09-18T15:41:09.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:41:09.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:41:09.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:41:09.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:41:09.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:41:09.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:41:09.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:41:09.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:41:09.322 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:41:09.322 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:41:09.323 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:41:09.323 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:41:09.323 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:41:09.323 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:41:09.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:41:09.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:41:09.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:41:09.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:41:09.541 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:41:09.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:41:09.897 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:41:09.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:41:09.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:41:09.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:41:10.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836505 2024-09-18T15:41:10.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836505 2024-09-18T15:41:10.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505' 2024-09-18T15:41:10.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:41:10.039 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:41:10.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=292057776130 2024-09-18T15:41:10.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 292057776130 2024-09-18T15:41:10.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-292057776130' 2024-09-18T15:41:10.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:41:10.184 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836505 2024-09-18T15:41:10.184 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:41:10.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:41:10.188 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836505 2024-09-18T15:41:10.188 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:41:10.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836505 2024-09-18T15:41:10.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-18T15:41:10.190 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836505 2024-09-18T15:41:10.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:41:10.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836504 -lt 21474836505 2024-09-18T15:41:10.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:41:11.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:41:11.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:41:11.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836506 -lt 21474836505 2024-09-18T15:41:11.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:41:11.853 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-292057776130 2024-09-18T15:41:11.853 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:41:11.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:41:11.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-292057776130 2024-09-18T15:41:11.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:41:11.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=292057776130 2024-09-18T15:41:11.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 292057776130' 2024-09-18T15:41:11.860 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 292057776130 2024-09-18T15:41:11.860 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:41:12.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 292057776130 -lt 292057776130 2024-09-18T15:41:12.206 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:41:12.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:41:12.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:41:12.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:41:12.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:41:12.625 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:41:12.625 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:41:12.625 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:41:12.625 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:41:12.625 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:41:12.626 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:41:12.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:41:12.968 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:41:12.968 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:41:12.968 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:41:13.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:41:13.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:41:13.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:41:13.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:41:13.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-09-18T15:41:13.359 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: expr 2 % 2 2024-09-18T15:41:13.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: local osd=0 2024-09-18T15:41:13.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: case $i in 2024-09-18T15:41:13.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1062: TEST_corrupt_scrub_replicated: local payload=UVWXYZ 2024-09-18T15:41:13.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1063: TEST_corrupt_scrub_replicated: echo UVWXYZ 2024-09-18T15:41:13.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1064: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:41:13.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:41:13.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:41:13.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:41:13.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:41:13.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:41:13.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:41:13.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:41:13.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:41:13.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:41:13.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:41:13.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:41:13.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:41:13.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:41:13.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:41:13.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:41:13.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:41:13.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:41:13.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:41:13.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:41:13.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:41:13.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:41:13.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:41:13.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:41:15.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-18T15:41:15.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:41:15.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:41:15.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:41:15.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:41:15.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:41:15.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:41:15.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:41:15.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:41:15.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:41:15.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:41:15.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:41:15.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:41:15.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:41:15.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:41:15.853 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:41:15.853 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:41:15.853 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:41:15.853 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:41:15.853 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:41:15.853 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:41:15.853 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:41:15.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:41:15.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:41:15.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:41:15.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:41:15.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:41:15.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:41:15.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:41:15.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:41:15.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:41:15.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:41:15.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:41:15.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:41:15.856 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:41:15.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:41:15.858 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:41:15.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:41:15.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:41:15.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:41:15.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:41:15.888 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:41:15.880+0000 7f665d379540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:41:15.891 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:41:15.880+0000 7f665d379540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:41:15.893 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:41:15.884+0000 7f665d379540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:41:16.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:41:16.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:41:16.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:41:16.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:41:16.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:41:16.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:16.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:41:16.176 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:41:16.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:16.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:41:16.439 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:41:16.428+0000 7f665d379540 -1 Falling back to public interface 2024-09-18T15:41:16.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:17.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:17.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:17.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:41:17.538 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:41:17.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:17.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:41:17.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:18.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:18.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:18.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:41:18.877 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:41:18.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:18.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:41:19.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:19.299 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:41:19.288+0000 7f665d379540 -1 osd.0 70 log_to_monitors true 2024-09-18T15:41:20.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:20.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:20.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:41:20.236 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:41:20.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:20.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:41:20.679 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 74 up_thru 74 down_at 71 last_clean_interval [5,70) [v2:127.0.0.1:6802/365606,v1:127.0.0.1:6803/365606] [v2:127.0.0.1:6804/365606,v1:127.0.0.1:6805/365606] exists,up 537f9958-5809-4e44-8060-20ee457d6502 2024-09-18T15:41:20.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:41:20.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:41:20.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:41:20.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:41:20.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:41:20.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:41:20.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:41:20.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:41:20.680 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:41:20.680 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:41:20.681 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:41:20.681 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:41:20.681 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:41:20.681 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:41:20.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:41:20.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:41:20.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:41:20.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:41:20.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:41:21.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:41:21.226 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:41:21.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:41:21.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:41:21.226 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:41:21.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=317827579906 2024-09-18T15:41:21.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 317827579906 2024-09-18T15:41:21.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-317827579906' 2024-09-18T15:41:21.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:41:21.399 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:41:21.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=292057776133 2024-09-18T15:41:21.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 292057776133 2024-09-18T15:41:21.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-317827579906 1-292057776133' 2024-09-18T15:41:21.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:41:21.536 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-317827579906 2024-09-18T15:41:21.537 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:41:21.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:41:21.540 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-317827579906 2024-09-18T15:41:21.540 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:41:21.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=317827579906 2024-09-18T15:41:21.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 317827579906' 2024-09-18T15:41:21.542 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 317827579906 2024-09-18T15:41:21.543 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:41:21.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 317827579906 2024-09-18T15:41:21.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:41:22.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:41:22.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:41:23.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 317827579906 -lt 317827579906 2024-09-18T15:41:23.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:41:23.192 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-292057776133 2024-09-18T15:41:23.192 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:41:23.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:41:23.195 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-292057776133 2024-09-18T15:41:23.196 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:41:23.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=292057776133 2024-09-18T15:41:23.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 292057776133' 2024-09-18T15:41:23.198 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 292057776133 2024-09-18T15:41:23.199 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:41:23.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 292057776133 -lt 292057776133 2024-09-18T15:41:23.528 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:41:23.529 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:41:23.530 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:41:23.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:41:23.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:41:23.909 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:41:23.909 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:41:23.909 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:41:23.909 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:41:23.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:41:23.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:41:24.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:41:24.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:41:24.219 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:41:24.219 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:41:24.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:41:24.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:41:24.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:41:24.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:41:24.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-09-18T15:41:24.590 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: expr 3 % 2 2024-09-18T15:41:24.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: local osd=1 2024-09-18T15:41:24.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: case $i in 2024-09-18T15:41:24.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ3 remove 2024-09-18T15:41:24.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:41:24.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:41:24.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:41:24.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:41:24.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ3 remove 2024-09-18T15:41:24.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:41:24.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:41:24.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:41:24.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:41:24.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:41:24.595 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:41:24.595 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:41:24.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:41:24.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:41:24.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:41:24.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:41:24.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ3 remove 2024-09-18T15:41:24.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:41:24.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:41:24.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:41:24.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:41:24.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:41:24.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ3 remove 2024-09-18T15:41:26.361 INFO:tasks.workunit.client.0.smithi046.stdout:remove #3:f2a5b2a4:::ROBJ3:head# 2024-09-18T15:41:26.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:41:26.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:41:26.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:41:26.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:41:26.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:41:26.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:41:26.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:41:26.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:41:26.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:41:26.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:41:26.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:41:26.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:41:26.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:41:26.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:41:26.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:41:26.905 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:41:26.905 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:41:26.905 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:41:26.905 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:41:26.905 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:41:26.906 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:41:26.906 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:41:26.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:41:26.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:41:26.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:41:26.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:41:26.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:41:26.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:41:26.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:41:26.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:41:26.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:41:26.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:41:26.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:41:26.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:41:26.908 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:41:26.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:41:26.909 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:41:26.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:41:26.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:41:26.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:41:26.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:41:26.940 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:41:26.932+0000 7fd028af4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:41:26.944 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:41:26.932+0000 7fd028af4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:41:26.945 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:41:26.936+0000 7fd028af4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:41:27.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:41:27.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:41:27.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:41:27.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:41:27.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:41:27.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:27.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:41:27.219 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:41:27.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:27.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:27.486 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:41:27.476+0000 7fd028af4540 -1 Falling back to public interface 2024-09-18T15:41:27.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:28.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:28.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:28.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:41:28.536 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:41:28.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:28.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:28.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:29.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:29.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:29.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:41:29.850 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:41:29.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:29.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:30.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:30.437 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:41:30.432+0000 7fd028af4540 -1 osd.1 75 log_to_monitors true 2024-09-18T15:41:31.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:31.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:31.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:41:31.166 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:41:31.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:31.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:31.539 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 79 up_thru 79 down_at 76 last_clean_interval [68,75) [v2:127.0.0.1:6810/366909,v1:127.0.0.1:6811/366909] [v2:127.0.0.1:6812/366909,v1:127.0.0.1:6813/366909] exists,up 8c8956ed-6b0c-4a8b-8558-811b867c0d90 2024-09-18T15:41:31.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:41:31.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:41:31.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:41:31.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:41:31.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:41:31.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:41:31.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:41:31.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:41:31.540 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:41:31.540 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:41:31.540 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:41:31.541 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:41:31.541 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:41:31.541 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:41:31.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:41:31.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:41:31.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:41:31.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:41:31.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:41:32.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:41:32.075 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:41:32.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:41:32.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:41:32.076 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:41:32.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=317827579909 2024-09-18T15:41:32.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 317827579909 2024-09-18T15:41:32.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-317827579909' 2024-09-18T15:41:32.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:41:32.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:41:32.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=339302416386 2024-09-18T15:41:32.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 339302416386 2024-09-18T15:41:32.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-317827579909 1-339302416386' 2024-09-18T15:41:32.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:41:32.393 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-317827579909 2024-09-18T15:41:32.393 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:41:32.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:41:32.397 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-317827579909 2024-09-18T15:41:32.397 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:41:32.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=317827579909 2024-09-18T15:41:32.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 317827579909' 2024-09-18T15:41:32.399 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 317827579909 2024-09-18T15:41:32.399 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:41:32.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 317827579908 -lt 317827579909 2024-09-18T15:41:32.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:41:33.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:41:33.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:41:34.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 317827579909 -lt 317827579909 2024-09-18T15:41:34.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:41:34.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-339302416386 2024-09-18T15:41:34.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:41:34.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:41:34.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-339302416386 2024-09-18T15:41:34.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:41:34.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=339302416386 2024-09-18T15:41:34.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 339302416386' 2024-09-18T15:41:34.072 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 339302416386 2024-09-18T15:41:34.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:41:34.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 339302416386 -lt 339302416386 2024-09-18T15:41:34.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:41:34.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:41:34.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:41:34.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:41:34.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:41:34.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:41:34.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:41:34.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:41:34.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:41:34.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:41:34.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:41:35.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:41:35.166 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:41:35.166 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:41:35.166 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:41:35.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:41:35.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:41:35.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:41:35.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:41:35.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-09-18T15:41:35.696 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: expr 4 % 2 2024-09-18T15:41:35.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: local osd=0 2024-09-18T15:41:35.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: case $i in 2024-09-18T15:41:35.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1074: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-09-18T15:41:35.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:41:35.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:41:35.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:41:35.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:41:35.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-09-18T15:41:35.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:41:35.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:41:35.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:41:35.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:41:35.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:41:35.699 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:41:35.699 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:41:35.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:41:35.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:41:35.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:41:35.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:41:35.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-09-18T15:41:35.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:41:35.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:41:35.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:41:35.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:41:35.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:41:35.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-09-18T15:41:37.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-18T15:41:37.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:41:37.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:41:37.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:41:37.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:41:37.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:41:37.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:41:37.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:41:37.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:41:37.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:41:37.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:41:37.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:41:37.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:41:37.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:41:37.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:41:37.877 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:41:37.877 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:41:37.877 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:41:37.877 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:41:37.877 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:41:37.877 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:41:37.878 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:41:37.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:41:37.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:41:37.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:41:37.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:41:37.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:41:37.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:41:37.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:41:37.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:41:37.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:41:37.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:41:37.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:41:37.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:41:37.880 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:41:37.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:41:37.881 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:41:37.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:41:37.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:41:37.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:41:37.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:41:37.914 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:41:37.908+0000 7f8bb1de6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:41:37.938 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:41:37.932+0000 7f8bb1de6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:41:37.939 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:41:37.932+0000 7f8bb1de6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:41:38.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:41:38.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:41:38.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:41:38.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:41:38.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:41:38.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:38.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:41:38.188 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:41:38.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:38.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:41:38.479 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:41:38.472+0000 7f8bb1de6540 -1 Falling back to public interface 2024-09-18T15:41:38.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:39.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:39.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:39.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:41:39.549 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:41:39.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:39.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:41:39.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:40.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:40.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:40.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:41:40.870 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:41:40.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:40.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:41:41.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:41.402 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:41:41.396+0000 7f8bb1de6540 -1 osd.0 80 log_to_monitors true 2024-09-18T15:41:42.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:42.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:42.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:41:42.210 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:41:42.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:42.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:41:42.626 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 84 up_thru 84 down_at 81 last_clean_interval [74,80) [v2:127.0.0.1:6802/368212,v1:127.0.0.1:6803/368212] [v2:127.0.0.1:6804/368212,v1:127.0.0.1:6805/368212] exists,up 537f9958-5809-4e44-8060-20ee457d6502 2024-09-18T15:41:42.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:41:42.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:41:42.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:41:42.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:41:42.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:41:42.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:41:42.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:41:42.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:41:42.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:41:42.629 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:41:42.629 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:41:42.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:41:42.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:41:42.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:41:42.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:41:42.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:41:42.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:41:42.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:41:42.849 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:41:43.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:41:43.155 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:41:43.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:41:43.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:41:43.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:41:43.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=360777252866 2024-09-18T15:41:43.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 360777252866 2024-09-18T15:41:43.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-360777252866' 2024-09-18T15:41:43.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:41:43.303 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:41:43.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=339302416389 2024-09-18T15:41:43.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 339302416389 2024-09-18T15:41:43.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-360777252866 1-339302416389' 2024-09-18T15:41:43.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:41:43.442 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-360777252866 2024-09-18T15:41:43.442 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:41:43.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:41:43.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-360777252866 2024-09-18T15:41:43.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:41:43.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=360777252866 2024-09-18T15:41:43.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 360777252866' 2024-09-18T15:41:43.448 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 360777252866 2024-09-18T15:41:43.449 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:41:43.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 360777252866 2024-09-18T15:41:43.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:41:44.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:41:44.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:41:45.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 360777252866 2024-09-18T15:41:45.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:41:46.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-18T15:41:46.111 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:41:46.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 360777252866 -lt 360777252866 2024-09-18T15:41:46.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:41:46.419 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-339302416389 2024-09-18T15:41:46.419 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:41:46.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:41:46.423 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-339302416389 2024-09-18T15:41:46.423 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:41:46.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=339302416389 2024-09-18T15:41:46.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 339302416389' 2024-09-18T15:41:46.424 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 339302416389 2024-09-18T15:41:46.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:41:46.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 339302416389 -lt 339302416389 2024-09-18T15:41:46.763 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:41:46.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:41:46.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:41:47.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:41:47.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:41:47.177 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:41:47.177 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:41:47.178 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:41:47.178 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:41:47.178 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:41:47.178 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:41:47.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:41:47.530 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:41:47.531 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:41:47.531 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:41:47.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:41:47.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:41:47.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:41:47.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:41:47.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-09-18T15:41:47.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: expr 5 % 2 2024-09-18T15:41:47.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: local osd=1 2024-09-18T15:41:47.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: case $i in 2024-09-18T15:41:47.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1079: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-09-18T15:41:47.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:41:47.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:41:47.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:41:47.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:41:47.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-09-18T15:41:47.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:41:47.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:41:47.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:41:47.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:41:47.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:41:47.893 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:41:47.893 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:41:47.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:41:47.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:41:47.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:41:48.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:41:48.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-09-18T15:41:48.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:41:48.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:41:48.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:41:48.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:41:48.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:41:48.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ5 rm-omap key-ROBJ5 2024-09-18T15:41:50.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:41:50.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:41:50.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:41:50.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:41:50.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:41:50.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:41:50.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:41:50.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:41:50.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:41:50.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:41:50.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:41:50.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:41:50.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:41:50.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:41:50.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:41:50.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:41:50.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:41:50.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:41:50.481 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:41:50.482 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:41:50.482 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:41:50.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:41:50.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:41:50.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:41:50.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:41:50.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:41:50.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:41:50.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:41:50.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:41:50.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:41:50.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:41:50.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:41:50.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:41:50.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:41:50.485 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:41:50.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:41:50.490 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:41:50.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:41:50.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:41:50.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:41:50.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:41:50.518 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:41:50.511+0000 7f395e104540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:41:50.521 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:41:50.515+0000 7f395e104540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:41:50.524 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:41:50.515+0000 7f395e104540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:41:50.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:41:50.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:41:50.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:41:50.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:41:50.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:41:50.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:50.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:41:50.796 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:41:50.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:50.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:51.059 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:41:51.051+0000 7f395e104540 -1 Falling back to public interface 2024-09-18T15:41:51.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:52.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:52.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:52.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:41:52.158 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:41:52.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:52.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:52.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:53.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:53.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:53.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:41:53.478 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:41:53.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:53.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:53.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:53.853 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:41:53.847+0000 7f395e104540 -1 osd.1 86 log_to_monitors true 2024-09-18T15:41:54.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:54.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:54.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:41:54.829 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:41:54.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:54.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:55.191 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 90 up_thru 85 down_at 87 last_clean_interval [79,86) [v2:127.0.0.1:6810/369539,v1:127.0.0.1:6811/369539] [v2:127.0.0.1:6812/369539,v1:127.0.0.1:6813/369539] exists,up 8c8956ed-6b0c-4a8b-8558-811b867c0d90 2024-09-18T15:41:55.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:41:55.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:41:55.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:41:55.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:41:55.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:41:55.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:41:55.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:41:55.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:41:55.192 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:41:55.193 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:41:55.193 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:41:55.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:41:55.194 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:41:55.194 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:41:55.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:41:55.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:41:55.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:41:55.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:41:55.427 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:41:55.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:41:55.777 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:41:55.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:41:55.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:41:55.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:41:55.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=360777252869 2024-09-18T15:41:55.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 360777252869 2024-09-18T15:41:55.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-360777252869' 2024-09-18T15:41:55.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:41:55.958 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:41:56.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=386547056642 2024-09-18T15:41:56.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 386547056642 2024-09-18T15:41:56.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-360777252869 1-386547056642' 2024-09-18T15:41:56.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:41:56.098 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-360777252869 2024-09-18T15:41:56.098 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:41:56.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:41:56.102 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-360777252869 2024-09-18T15:41:56.102 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:41:56.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=360777252869 2024-09-18T15:41:56.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 360777252869' 2024-09-18T15:41:56.105 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 360777252869 2024-09-18T15:41:56.106 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:41:56.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 360777252868 -lt 360777252869 2024-09-18T15:41:56.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:41:57.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:41:57.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:41:57.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 360777252870 -lt 360777252869 2024-09-18T15:41:57.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:41:57.732 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-386547056642 2024-09-18T15:41:57.732 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:41:57.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:41:57.736 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-386547056642 2024-09-18T15:41:57.736 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:41:57.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=386547056642 2024-09-18T15:41:57.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 386547056642' 2024-09-18T15:41:57.738 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 386547056642 2024-09-18T15:41:57.739 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:41:58.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 386547056642 -lt 386547056642 2024-09-18T15:41:58.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:41:58.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:41:58.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:41:58.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:41:58.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:41:58.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:41:58.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:41:58.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:41:58.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:41:58.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:41:58.494 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:41:58.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:41:58.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:41:58.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:41:58.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:41:59.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:41:59.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:41:59.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:41:59.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:41:59.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-09-18T15:41:59.215 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: expr 6 % 2 2024-09-18T15:41:59.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: local osd=0 2024-09-18T15:41:59.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: case $i in 2024-09-18T15:41:59.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1084: TEST_corrupt_scrub_replicated: echo extra 2024-09-18T15:41:59.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1085: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-09-18T15:41:59.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:41:59.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:41:59.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:41:59.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:41:59.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-09-18T15:41:59.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:41:59.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:41:59.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:41:59.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:41:59.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:41:59.219 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:41:59.219 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:41:59.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:41:59.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:41:59.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:41:59.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:41:59.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-09-18T15:41:59.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:41:59.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:41:59.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:41:59.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:41:59.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:41:59.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-09-18T15:42:01.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-18T15:42:01.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:42:01.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:42:01.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:42:01.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:42:01.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:42:01.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:42:01.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:42:01.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:42:01.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:42:01.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:42:01.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:42:01.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:42:01.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:42:01.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:42:01.401 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:42:01.401 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:42:01.401 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:42:01.401 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:42:01.401 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:42:01.401 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:42:01.401 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:42:01.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:42:01.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:42:01.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:42:01.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:42:01.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:42:01.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:42:01.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:42:01.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:42:01.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:42:01.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:42:01.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:42:01.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:42:01.404 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:42:01.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:42:01.406 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:42:01.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:42:01.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:42:01.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:42:01.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:42:01.437 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:01.431+0000 7f7918e26540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:42:01.440 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:01.435+0000 7f7918e26540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:42:01.442 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:01.435+0000 7f7918e26540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:42:01.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:42:01.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:42:01.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:42:01.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:42:01.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:42:01.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:01.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:42:01.713 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:42:01.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:01.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:42:01.983 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:01.975+0000 7f7918e26540 -1 Falling back to public interface 2024-09-18T15:42:02.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:03.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:03.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:03.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:42:03.054 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:42:03.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:03.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:42:03.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:04.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:04.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:04.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:42:04.375 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:42:04.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:04.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:42:04.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:04.724 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:04.719+0000 7f7918e26540 -1 osd.0 92 log_to_monitors true 2024-09-18T15:42:05.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:05.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:05.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:42:05.693 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:42:05.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:05.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:42:06.050 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 96 up_thru 90 down_at 93 last_clean_interval [84,92) [v2:127.0.0.1:6802/370842,v1:127.0.0.1:6803/370842] [v2:127.0.0.1:6804/370842,v1:127.0.0.1:6805/370842] exists,up 537f9958-5809-4e44-8060-20ee457d6502 2024-09-18T15:42:06.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:42:06.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:42:06.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:42:06.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:42:06.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:42:06.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:42:06.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:42:06.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:42:06.051 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:42:06.052 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:42:06.052 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:42:06.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:42:06.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:42:06.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:42:06.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:42:06.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:42:06.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:42:06.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:42:06.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:42:06.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:42:06.635 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:42:06.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:42:06.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:06.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:42:06.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=412316860418 2024-09-18T15:42:06.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 412316860418 2024-09-18T15:42:06.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-412316860418' 2024-09-18T15:42:06.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:06.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:42:06.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=386547056645 2024-09-18T15:42:06.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 386547056645 2024-09-18T15:42:06.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-412316860418 1-386547056645' 2024-09-18T15:42:06.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:06.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-412316860418 2024-09-18T15:42:06.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:06.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:42:06.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-412316860418 2024-09-18T15:42:06.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:06.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=412316860418 2024-09-18T15:42:06.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 412316860418' 2024-09-18T15:42:06.926 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 412316860418 2024-09-18T15:42:06.926 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:42:07.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 412316860418 -lt 412316860418 2024-09-18T15:42:07.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:07.247 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-386547056645 2024-09-18T15:42:07.247 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:07.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:42:07.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-386547056645 2024-09-18T15:42:07.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:07.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=386547056645 2024-09-18T15:42:07.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 386547056645' 2024-09-18T15:42:07.254 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 386547056645 2024-09-18T15:42:07.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:42:07.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 386547056645 -lt 386547056645 2024-09-18T15:42:07.577 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:42:07.578 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:42:07.578 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:42:07.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:42:07.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:42:07.950 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:42:07.950 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:42:07.950 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:42:07.950 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:42:07.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:42:07.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:42:08.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-09-18T15:42:08.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:42:08.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:42:08.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:42:08.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-09-18T15:42:08.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-09-18T15:42:08.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T15:42:08.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-09-18T15:42:08.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:42:08.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:42:08.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:42:08.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:42:08.717 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:42:08.717 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:42:08.717 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:42:08.718 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:42:08.718 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:42:08.718 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:42:09.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-09-18T15:42:09.033 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:42:09.033 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:42:09.033 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:42:09.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-09-18T15:42:09.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-09-18T15:42:09.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:42:09.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:42:09.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:42:09.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:42:09.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:42:09.391 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:42:09.392 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:42:09.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T15:42:09.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T15:42:09.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-09-18T15:42:09.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:42:09.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-18T15:42:09.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:42:09.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:42:09.958 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:42:09.958 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:42:09.958 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:42:09.958 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:42:09.959 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:42:09.959 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:42:10.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-09-18T15:42:10.269 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:42:10.270 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:42:10.270 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:42:10.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-09-18T15:42:10.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-09-18T15:42:10.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:42:10.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:42:10.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:42:10.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:42:10.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:42:10.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:42:10.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:42:10.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T15:42:10.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T15:42:10.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-09-18T15:42:10.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:42:10.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-18T15:42:11.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:42:11.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:42:11.381 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:42:11.381 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:42:11.381 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:42:11.381 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:42:11.381 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:42:11.381 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:42:11.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:42:11.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:42:11.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:42:11.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:42:12.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:42:12.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:42:12.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:42:12.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1086: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/extra-val 2024-09-18T15:42:12.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:42:12.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-09-18T15:42:12.063 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: expr 7 % 2 2024-09-18T15:42:12.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: local osd=1 2024-09-18T15:42:12.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: case $i in 2024-09-18T15:42:12.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1091: TEST_corrupt_scrub_replicated: echo -n newheader 2024-09-18T15:42:12.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-09-18T15:42:12.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:42:12.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:42:12.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:42:12.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:42:12.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-09-18T15:42:12.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:42:12.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:42:12.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:42:12.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:42:12.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:42:12.067 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:42:12.067 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:42:12.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:42:12.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:42:12.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:42:12.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:42:12.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-09-18T15:42:12.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:42:12.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:42:12.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:42:12.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:42:12.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:42:12.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-09-18T15:42:14.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:42:14.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:42:14.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:42:14.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:42:14.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:42:14.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:42:14.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:42:14.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:42:14.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:42:14.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:42:14.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:42:14.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:42:14.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:42:14.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:42:14.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:42:14.449 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:42:14.449 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:42:14.449 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:42:14.449 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:42:14.449 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:42:14.449 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:42:14.449 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:42:14.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:42:14.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:42:14.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:42:14.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:42:14.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:42:14.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:42:14.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:42:14.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:42:14.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:42:14.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:42:14.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:42:14.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:42:14.452 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:42:14.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:42:14.454 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:42:14.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:42:14.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:42:14.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:42:14.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:42:14.484 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:14.479+0000 7fa66469d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:42:14.489 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:14.483+0000 7fa66469d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:42:14.495 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:14.483+0000 7fa66469d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:42:14.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:42:14.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:42:14.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:42:14.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:42:14.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:42:14.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:14.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:42:14.772 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:42:14.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:14.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:42:15.023 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:15.015+0000 7fa66469d540 -1 Falling back to public interface 2024-09-18T15:42:15.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:16.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:16.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:16.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:42:16.129 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:42:16.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:16.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:42:16.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:17.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:17.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:17.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:42:17.453 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:42:17.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:17.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:42:17.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:17.897 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:17.891+0000 7fa66469d540 -1 osd.1 98 log_to_monitors true 2024-09-18T15:42:18.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:18.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:18.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:42:18.771 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:42:18.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:18.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:42:19.246 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 102 up_thru 102 down_at 99 last_clean_interval [90,98) [v2:127.0.0.1:6810/372325,v1:127.0.0.1:6811/372325] [v2:127.0.0.1:6812/372325,v1:127.0.0.1:6813/372325] exists,up 8c8956ed-6b0c-4a8b-8558-811b867c0d90 2024-09-18T15:42:19.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:42:19.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:42:19.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:42:19.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:42:19.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:42:19.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:42:19.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:42:19.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:42:19.247 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:42:19.248 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:42:19.248 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:42:19.248 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:42:19.248 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:42:19.248 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:42:19.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:42:19.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:42:19.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:42:19.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:42:19.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:42:19.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:42:19.796 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:42:19.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:42:19.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:19.797 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:42:19.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=412316860421 2024-09-18T15:42:19.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 412316860421 2024-09-18T15:42:19.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-412316860421' 2024-09-18T15:42:19.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:19.970 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:42:20.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=438086664194 2024-09-18T15:42:20.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 438086664194 2024-09-18T15:42:20.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-412316860421 1-438086664194' 2024-09-18T15:42:20.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:20.113 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-412316860421 2024-09-18T15:42:20.113 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:20.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:42:20.117 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-412316860421 2024-09-18T15:42:20.117 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:20.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=412316860421 2024-09-18T15:42:20.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 412316860421' 2024-09-18T15:42:20.120 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 412316860421 2024-09-18T15:42:20.120 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:42:20.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 412316860420 -lt 412316860421 2024-09-18T15:42:20.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:42:21.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:42:21.423 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:42:21.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 412316860422 -lt 412316860421 2024-09-18T15:42:21.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:21.746 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-438086664194 2024-09-18T15:42:21.746 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:21.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:42:21.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-438086664194 2024-09-18T15:42:21.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:21.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=438086664194 2024-09-18T15:42:21.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 438086664194' 2024-09-18T15:42:21.752 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 438086664194 2024-09-18T15:42:21.753 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:42:22.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 438086664194 -lt 438086664194 2024-09-18T15:42:22.067 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:42:22.067 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:42:22.067 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:42:22.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:42:22.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:42:22.436 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:42:22.436 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:42:22.436 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:42:22.436 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:42:22.437 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:42:22.437 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:42:22.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:42:22.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:42:22.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:42:22.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:42:23.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:42:23.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:42:23.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:42:23.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1093: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/hdr 2024-09-18T15:42:23.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:42:23.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-09-18T15:42:23.158 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: expr 8 % 2 2024-09-18T15:42:23.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: local osd=0 2024-09-18T15:42:23.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: case $i in 2024-09-18T15:42:23.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1097: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key1-ROBJ8 val1-ROBJ8 2024-09-18T15:42:23.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1098: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key2-ROBJ8 val2-ROBJ8 2024-09-18T15:42:23.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1101: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-09-18T15:42:23.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1102: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-09-18T15:42:23.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:42:23.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:42:23.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:42:23.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:42:23.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-09-18T15:42:23.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:42:23.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:42:23.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:42:23.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:42:23.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:42:23.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:42:23.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:42:23.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:42:23.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:42:23.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:42:23.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:42:23.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-09-18T15:42:23.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:42:23.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:42:23.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:42:23.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:42:23.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:42:23.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-09-18T15:42:25.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-18T15:42:25.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:42:25.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:42:25.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:42:25.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:42:25.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:42:25.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:42:25.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:42:25.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:42:25.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:42:25.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:42:25.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:42:25.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:42:25.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:42:25.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:42:25.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:42:25.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:42:25.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:42:25.414 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:42:25.414 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:42:25.414 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:42:25.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:42:25.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:42:25.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:42:25.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:42:25.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:42:25.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:42:25.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:42:25.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:42:25.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:42:25.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:42:25.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:42:25.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:42:25.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:42:25.417 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:42:25.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:42:25.418 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:42:25.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:42:25.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:42:25.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:42:25.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:42:25.450 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:25.443+0000 7fcfaed6b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:42:25.452 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:25.447+0000 7fcfaed6b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:42:25.454 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:25.447+0000 7fcfaed6b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:42:25.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:42:25.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:42:25.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:42:25.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:42:25.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:42:25.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:25.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:42:25.725 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:42:25.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:25.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:42:25.995 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:25.987+0000 7fcfaed6b540 -1 Falling back to public interface 2024-09-18T15:42:26.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:27.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:27.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:27.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:42:27.074 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:42:27.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:27.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:42:27.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:28.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:28.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:28.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:42:28.380 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:42:28.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:28.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:42:28.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:28.912 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:28.907+0000 7fcfaed6b540 -1 osd.0 103 log_to_monitors true 2024-09-18T15:42:29.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:29.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:29.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:42:29.698 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:42:29.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:29.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:42:30.146 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 107 up_thru 107 down_at 104 last_clean_interval [96,103) [v2:127.0.0.1:6802/373669,v1:127.0.0.1:6803/373669] [v2:127.0.0.1:6804/373669,v1:127.0.0.1:6805/373669] exists,up 537f9958-5809-4e44-8060-20ee457d6502 2024-09-18T15:42:30.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:42:30.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:42:30.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:42:30.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:42:30.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:42:30.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:42:30.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:42:30.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:42:30.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:42:30.149 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:42:30.150 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:42:30.150 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:42:30.150 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:42:30.150 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:42:30.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:42:30.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:42:30.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:42:30.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:42:30.365 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:42:30.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:42:30.703 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:42:30.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:42:30.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:30.704 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:42:30.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=459561500674 2024-09-18T15:42:30.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 459561500674 2024-09-18T15:42:30.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-459561500674' 2024-09-18T15:42:30.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:30.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:42:31.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=438086664197 2024-09-18T15:42:31.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 438086664197 2024-09-18T15:42:31.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-459561500674 1-438086664197' 2024-09-18T15:42:31.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:31.027 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-459561500674 2024-09-18T15:42:31.027 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:31.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:42:31.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-459561500674 2024-09-18T15:42:31.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:31.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=459561500674 2024-09-18T15:42:31.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 459561500674' 2024-09-18T15:42:31.033 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 459561500674 2024-09-18T15:42:31.033 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:42:31.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 459561500674 -lt 459561500674 2024-09-18T15:42:31.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:31.353 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-438086664197 2024-09-18T15:42:31.353 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:31.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:42:31.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-438086664197 2024-09-18T15:42:31.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:31.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=438086664197 2024-09-18T15:42:31.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 438086664197' 2024-09-18T15:42:31.360 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 438086664197 2024-09-18T15:42:31.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:42:31.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 438086664197 -lt 438086664197 2024-09-18T15:42:31.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:42:31.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:42:31.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:42:32.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:42:32.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:42:32.104 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:42:32.104 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:42:32.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:42:32.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:42:32.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:42:32.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:42:32.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:42:32.444 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:42:32.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:42:32.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:42:32.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:42:32.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:42:32.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:42:32.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1103: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-09-18T15:42:32.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:42:32.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:42:32.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:42:32.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:42:32.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-09-18T15:42:32.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:42:32.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:42:32.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:42:32.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:42:32.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:42:32.824 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:42:32.824 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:42:32.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:42:32.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:42:32.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:42:33.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:42:33.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-09-18T15:42:33.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:42:33.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:42:33.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:42:33.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:42:33.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:42:33.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 rm-attr _key2-ROBJ8 2024-09-18T15:42:34.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:42:34.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:42:34.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:42:34.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:42:34.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:42:34.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:42:34.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:42:34.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:42:34.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:42:34.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:42:34.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:42:34.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:42:34.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:42:34.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:42:34.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:42:34.980 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:42:34.980 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:42:34.980 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:42:34.980 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:42:34.981 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:42:34.981 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:42:34.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:42:34.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:42:34.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:42:34.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:42:34.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:42:34.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:42:34.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:42:34.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:42:34.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:42:34.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:42:34.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:42:34.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:42:34.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:42:34.984 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:42:34.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:42:34.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:42:34.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:42:34.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:42:34.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:42:34.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:42:35.016 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:35.007+0000 7f9b6e078540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:42:35.019 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:35.011+0000 7f9b6e078540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:42:35.021 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:35.015+0000 7f9b6e078540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:42:35.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:42:35.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:42:35.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:42:35.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:42:35.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:42:35.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:35.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:42:35.327 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:42:35.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:35.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:42:35.554 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:35.547+0000 7f9b6e078540 -1 Falling back to public interface 2024-09-18T15:42:35.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:36.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:36.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:36.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:42:36.675 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:42:36.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:36.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:42:36.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:37.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:37.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:37.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:42:37.986 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:42:37.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:37.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:42:38.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:38.646 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:38.639+0000 7f9b6e078540 -1 osd.1 109 log_to_monitors true 2024-09-18T15:42:39.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:39.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:39.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:42:39.304 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:42:39.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:39.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:42:39.693 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 113 up_thru 113 down_at 110 last_clean_interval [102,109) [v2:127.0.0.1:6810/374947,v1:127.0.0.1:6811/374947] [v2:127.0.0.1:6812/374947,v1:127.0.0.1:6813/374947] exists,up 8c8956ed-6b0c-4a8b-8558-811b867c0d90 2024-09-18T15:42:39.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:42:39.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:42:39.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:42:39.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:42:39.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:42:39.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:42:39.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:42:39.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:42:39.695 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:42:39.696 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:42:39.696 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:42:39.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:42:39.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:42:39.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:42:39.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:42:39.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:42:39.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:42:39.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:42:39.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:42:40.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:42:40.305 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:42:40.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:42:40.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:40.305 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:42:40.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=459561500677 2024-09-18T15:42:40.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 459561500677 2024-09-18T15:42:40.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-459561500677' 2024-09-18T15:42:40.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:40.485 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:42:40.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=485331304450 2024-09-18T15:42:40.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 485331304450 2024-09-18T15:42:40.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-459561500677 1-485331304450' 2024-09-18T15:42:40.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:40.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:40.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-459561500677 2024-09-18T15:42:40.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:42:40.676 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-459561500677 2024-09-18T15:42:40.677 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:40.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=459561500677 2024-09-18T15:42:40.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 459561500677' 2024-09-18T15:42:40.679 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 459561500677 2024-09-18T15:42:40.680 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:42:41.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 459561500675 -lt 459561500677 2024-09-18T15:42:41.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:42:42.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:42:42.005 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:42:42.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 459561500677 -lt 459561500677 2024-09-18T15:42:42.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:42.327 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-485331304450 2024-09-18T15:42:42.327 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:42.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:42:42.331 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-485331304450 2024-09-18T15:42:42.331 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:42.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=485331304450 2024-09-18T15:42:42.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 485331304450' 2024-09-18T15:42:42.334 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 485331304450 2024-09-18T15:42:42.335 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:42:42.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 485331304450 -lt 485331304450 2024-09-18T15:42:42.679 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:42:42.679 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:42:42.680 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:42:43.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:42:43.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:42:43.093 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:42:43.093 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:42:43.093 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:42:43.093 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:42:43.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:42:43.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:42:43.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:42:43.450 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:42:43.450 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:42:43.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:42:43.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:42:43.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:42:43.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:42:43.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1104: TEST_corrupt_scrub_replicated: echo -n val3-ROBJ8 2024-09-18T15:42:43.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1105: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-09-18T15:42:43.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:42:43.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:42:43.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:42:43.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:42:43.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-09-18T15:42:43.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:42:43.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:42:43.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:42:43.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:42:43.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:42:43.814 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:42:43.814 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:42:43.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:42:43.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:42:43.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:42:45.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:42:45.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-09-18T15:42:45.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:42:45.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:42:45.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:42:45.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:42:45.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:42:45.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-09-18T15:42:46.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:42:46.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:42:46.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:42:46.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:42:46.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:42:46.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:42:46.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:42:46.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:42:46.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:42:46.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:42:46.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:42:46.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:42:46.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:42:46.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:42:46.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:42:46.976 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:42:46.976 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:42:46.977 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:42:46.977 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:42:46.977 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:42:46.977 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:42:46.977 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:42:46.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:42:46.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:42:46.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:42:46.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:42:46.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:42:46.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:42:46.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:42:46.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:42:46.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:42:46.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:42:46.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:42:46.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:42:46.980 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:42:46.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:42:46.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:42:46.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:42:46.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:42:46.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:42:46.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:42:47.012 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:47.003+0000 7f4167a4e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:42:47.015 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:47.007+0000 7f4167a4e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:42:47.016 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:47.011+0000 7f4167a4e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:42:47.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:42:47.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:42:47.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:42:47.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:42:47.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:42:47.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:47.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:42:47.328 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:42:47.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:47.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:42:47.567 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:47.559+0000 7f4167a4e540 -1 Falling back to public interface 2024-09-18T15:42:47.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:48.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:48.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:48.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:42:48.621 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:42:48.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:48.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:42:48.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:49.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:49.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:49.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:42:49.932 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:42:49.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:49.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:42:50.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:50.448 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:50.439+0000 7f4167a4e540 -1 osd.1 115 log_to_monitors true 2024-09-18T15:42:51.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:51.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:51.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:42:51.249 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:42:51.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:51.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:42:51.727 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 119 up_thru 119 down_at 116 last_clean_interval [113,115) [v2:127.0.0.1:6810/376250,v1:127.0.0.1:6811/376250] [v2:127.0.0.1:6812/376250,v1:127.0.0.1:6813/376250] exists,up 8c8956ed-6b0c-4a8b-8558-811b867c0d90 2024-09-18T15:42:51.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:42:51.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:42:51.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:42:51.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:42:51.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:42:51.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:42:51.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:42:51.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:42:51.728 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:42:51.729 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:42:51.729 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:42:51.730 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:42:51.730 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:42:51.730 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:42:51.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:42:51.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:42:51.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:42:51.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:42:51.956 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:42:52.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:42:52.266 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:42:52.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:42:52.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:52.267 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:42:52.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=459561500680 2024-09-18T15:42:52.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 459561500680 2024-09-18T15:42:52.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-459561500680' 2024-09-18T15:42:52.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:52.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:42:52.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=511101108226 2024-09-18T15:42:52.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 511101108226 2024-09-18T15:42:52.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-459561500680 1-511101108226' 2024-09-18T15:42:52.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:52.594 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-459561500680 2024-09-18T15:42:52.594 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:52.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:42:52.598 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-459561500680 2024-09-18T15:42:52.598 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:52.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=459561500680 2024-09-18T15:42:52.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 459561500680' 2024-09-18T15:42:52.600 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 459561500680 2024-09-18T15:42:52.601 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:42:52.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 459561500679 -lt 459561500680 2024-09-18T15:42:52.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:42:53.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:42:53.919 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:42:54.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 459561500680 -lt 459561500680 2024-09-18T15:42:54.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:54.233 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-511101108226 2024-09-18T15:42:54.234 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:54.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:42:54.237 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-511101108226 2024-09-18T15:42:54.237 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:54.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=511101108226 2024-09-18T15:42:54.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 511101108226' 2024-09-18T15:42:54.240 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 511101108226 2024-09-18T15:42:54.240 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:42:54.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 511101108226 -lt 511101108226 2024-09-18T15:42:54.574 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:42:54.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:42:54.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:42:54.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:42:54.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:42:54.986 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:42:54.986 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:42:54.987 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:42:54.987 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:42:54.987 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:42:54.987 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:42:55.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:42:55.350 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:42:55.350 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:42:55.350 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:42:55.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:42:55.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:42:55.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:42:55.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1106: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-09-18T15:42:55.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:42:55.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-09-18T15:42:55.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: expr 9 % 2 2024-09-18T15:42:55.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: local osd=1 2024-09-18T15:42:55.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: case $i in 2024-09-18T15:42:55.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1110: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-09-18T15:42:55.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:42:55.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:42:55.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:42:55.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:42:55.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-09-18T15:42:55.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:42:55.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:42:55.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:42:55.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:42:55.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:42:55.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:42:55.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:42:55.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:42:55.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:42:55.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:42:56.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:42:56.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-09-18T15:42:56.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:42:56.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:42:56.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:42:56.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:42:56.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:42:56.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 get-attr _ 2024-09-18T15:42:58.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:42:58.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:42:58.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:42:58.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:42:58.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:42:58.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:42:58.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:42:58.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:42:58.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:42:58.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:42:58.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:42:58.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:42:58.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:42:58.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:42:58.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:42:58.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:42:58.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:42:58.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:42:58.073 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:42:58.073 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:42:58.073 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:42:58.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:42:58.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:42:58.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:42:58.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:42:58.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:42:58.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:42:58.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:42:58.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:42:58.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:42:58.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:42:58.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:42:58.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:42:58.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:42:58.076 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:42:58.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:42:58.078 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:42:58.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:42:58.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:42:58.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:42:58.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:42:58.109 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:58.102+0000 7f98f2518540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:42:58.113 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:58.106+0000 7f98f2518540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:42:58.121 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:58.114+0000 7f98f2518540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:42:58.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:42:58.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:42:58.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:42:58.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:42:58.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:42:58.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:58.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:42:58.385 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:42:58.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:58.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:42:58.659 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:42:58.650+0000 7f98f2518540 -1 Falling back to public interface 2024-09-18T15:42:58.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:59.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:59.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:59.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:42:59.733 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:42:59.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:59.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:43:00.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:43:01.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:43:01.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:43:01.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:43:01.090 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:43:01.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:43:01.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:43:01.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:43:01.583 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:43:01.574+0000 7f98f2518540 -1 osd.1 120 log_to_monitors true 2024-09-18T15:43:02.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:43:02.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:43:02.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:43:02.460 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:43:02.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:43:02.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:43:02.873 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 124 up_thru 124 down_at 121 last_clean_interval [119,120) [v2:127.0.0.1:6810/377554,v1:127.0.0.1:6811/377554] [v2:127.0.0.1:6812/377554,v1:127.0.0.1:6813/377554] exists,up 8c8956ed-6b0c-4a8b-8558-811b867c0d90 2024-09-18T15:43:02.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:43:02.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:43:02.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:43:02.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:43:02.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:43:02.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:43:02.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:43:02.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:43:02.874 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:43:02.876 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:43:02.876 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:43:02.876 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:43:02.876 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:43:02.876 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:43:03.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:43:03.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:43:03.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:43:03.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:43:03.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:43:03.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:43:03.473 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:43:03.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:43:03.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:43:03.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:43:03.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=459561500683 2024-09-18T15:43:03.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 459561500683 2024-09-18T15:43:03.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-459561500683' 2024-09-18T15:43:03.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:43:03.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:43:03.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=532575944706 2024-09-18T15:43:03.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 532575944706 2024-09-18T15:43:03.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-459561500683 1-532575944706' 2024-09-18T15:43:03.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:03.785 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-459561500683 2024-09-18T15:43:03.785 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:03.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:43:03.788 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-459561500683 2024-09-18T15:43:03.788 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:03.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=459561500683 2024-09-18T15:43:03.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 459561500683' 2024-09-18T15:43:03.791 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 459561500683 2024-09-18T15:43:03.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:43:04.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 459561500682 -lt 459561500683 2024-09-18T15:43:04.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:43:05.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:43:05.099 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:43:05.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 459561500684 -lt 459561500683 2024-09-18T15:43:05.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:05.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-532575944706 2024-09-18T15:43:05.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:05.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:43:05.415 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-532575944706 2024-09-18T15:43:05.415 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:05.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=532575944706 2024-09-18T15:43:05.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 532575944706' 2024-09-18T15:43:05.418 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 532575944706 2024-09-18T15:43:05.418 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:43:05.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 532575944706 -lt 532575944706 2024-09-18T15:43:05.758 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:43:05.758 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:43:05.759 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:43:06.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:43:06.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:43:06.131 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:43:06.132 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:43:06.132 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:43:06.132 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:43:06.132 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:43:06.132 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:43:06.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:43:06.486 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:43:06.487 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:43:06.487 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:43:06.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:43:06.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:43:06.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:43:06.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1111: TEST_corrupt_scrub_replicated: echo -n D 2024-09-18T15:43:06.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1112: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-09-18T15:43:06.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1113: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-18T15:43:06.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:43:06.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:43:06.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:43:06.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:43:06.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-18T15:43:06.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:43:06.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:43:06.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:43:06.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:43:06.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:43:06.990 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:43:06.990 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:43:06.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:43:06.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:43:06.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:43:08.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:43:08.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-18T15:43:08.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:43:08.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:43:08.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:43:08.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:43:08.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:43:08.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-18T15:43:10.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:43:10.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:43:10.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:43:10.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:43:10.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:43:10.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:43:10.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:43:10.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:43:10.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:43:10.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:43:10.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:43:10.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:43:10.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:43:10.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:43:10.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:43:10.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:43:10.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:43:10.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:43:10.182 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:43:10.182 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:43:10.182 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:43:10.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:43:10.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:43:10.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:43:10.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:43:10.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:43:10.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:43:10.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:43:10.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:43:10.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:43:10.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:43:10.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:43:10.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:43:10.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:43:10.185 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:43:10.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:43:10.186 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:43:10.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:43:10.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:43:10.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:43:10.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:43:10.217 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:43:10.210+0000 7f3af8c03540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:43:10.220 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:43:10.214+0000 7f3af8c03540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:43:10.224 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:43:10.214+0000 7f3af8c03540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:43:10.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:43:10.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:43:10.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:43:10.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:43:10.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:43:10.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:43:10.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:43:10.493 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:43:10.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:43:10.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:43:10.763 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:43:10.754+0000 7f3af8c03540 -1 Falling back to public interface 2024-09-18T15:43:10.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:43:11.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:43:11.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:43:11.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:43:11.843 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:43:11.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:43:11.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:43:12.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:43:13.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:43:13.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:43:13.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:43:13.160 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:43:13.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:43:13.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:43:13.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:43:13.499 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:43:13.490+0000 7f3af8c03540 -1 osd.1 126 log_to_monitors true 2024-09-18T15:43:14.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:43:14.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:43:14.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:43:14.484 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:43:14.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:43:14.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:43:14.928 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 130 up_thru 130 down_at 127 last_clean_interval [124,126) [v2:127.0.0.1:6810/378878,v1:127.0.0.1:6811/378878] [v2:127.0.0.1:6812/378878,v1:127.0.0.1:6813/378878] exists,up 8c8956ed-6b0c-4a8b-8558-811b867c0d90 2024-09-18T15:43:14.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:43:14.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:43:14.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:43:14.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:43:14.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:43:14.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:43:14.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:43:14.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:43:14.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:43:14.931 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:43:14.931 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:43:14.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:43:14.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:43:14.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:43:15.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:43:15.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:43:15.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:43:15.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:43:15.157 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:43:15.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:43:15.459 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:43:15.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:43:15.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:43:15.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:43:15.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=459561500687 2024-09-18T15:43:15.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 459561500687 2024-09-18T15:43:15.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-459561500687' 2024-09-18T15:43:15.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:43:15.637 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:43:15.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=558345748482 2024-09-18T15:43:15.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 558345748482 2024-09-18T15:43:15.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-459561500687 1-558345748482' 2024-09-18T15:43:15.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:15.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-459561500687 2024-09-18T15:43:15.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:15.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:43:15.781 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-459561500687 2024-09-18T15:43:15.782 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:15.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=459561500687 2024-09-18T15:43:15.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 459561500687' 2024-09-18T15:43:15.785 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 459561500687 2024-09-18T15:43:15.785 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:43:16.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 459561500686 -lt 459561500687 2024-09-18T15:43:16.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:43:17.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:43:17.128 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:43:17.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 459561500687 -lt 459561500687 2024-09-18T15:43:17.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:17.496 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-558345748482 2024-09-18T15:43:17.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:17.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:43:17.501 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-558345748482 2024-09-18T15:43:17.501 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:17.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=558345748482 2024-09-18T15:43:17.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 558345748482' 2024-09-18T15:43:17.503 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 558345748482 2024-09-18T15:43:17.504 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:43:17.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 558345748482 -lt 558345748482 2024-09-18T15:43:17.847 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:43:17.848 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:43:17.848 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:43:18.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:43:18.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:43:18.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:43:18.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:43:18.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:43:18.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:43:18.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:43:18.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:43:18.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:43:18.592 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:43:18.593 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:43:18.593 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:43:18.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:43:18.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:43:18.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:43:18.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1114: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-09-18T15:43:18.955 INFO:tasks.workunit.client.0.smithi046.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-09-18T15:43:18.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:43:18.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-09-18T15:43:18.956 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: expr 10 % 2 2024-09-18T15:43:18.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: local osd=0 2024-09-18T15:43:18.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: case $i in 2024-09-18T15:43:18.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:43:18.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-09-18T15:43:18.958 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: expr 11 % 2 2024-09-18T15:43:18.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: local osd=1 2024-09-18T15:43:18.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: case $i in 2024-09-18T15:43:18.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:43:18.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-09-18T15:43:18.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: expr 12 % 2 2024-09-18T15:43:18.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: local osd=0 2024-09-18T15:43:18.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: case $i in 2024-09-18T15:43:18.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:43:18.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-09-18T15:43:18.963 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: expr 13 % 2 2024-09-18T15:43:18.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: local osd=1 2024-09-18T15:43:18.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: case $i in 2024-09-18T15:43:18.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:43:18.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-09-18T15:43:18.965 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: expr 14 % 2 2024-09-18T15:43:18.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: local osd=0 2024-09-18T15:43:18.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: case $i in 2024-09-18T15:43:18.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1123: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-09-18T15:43:18.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1124: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-09-18T15:43:18.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:43:18.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:43:18.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:43:18.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:43:18.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-09-18T15:43:18.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:43:18.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:43:18.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:43:18.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:43:18.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:43:18.970 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:43:18.970 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:43:18.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:43:18.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:43:18.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:43:19.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:43:19.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-09-18T15:43:19.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:43:19.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:43:19.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:43:19.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:43:19.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:43:19.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-09-18T15:43:21.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-18T15:43:21.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:43:21.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:43:21.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:43:21.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:43:21.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:43:21.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:43:21.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:43:21.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:43:21.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:43:21.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:43:21.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:43:21.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:43:21.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:43:21.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:43:21.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:43:21.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:43:21.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:43:21.385 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:43:21.385 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:43:21.385 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:43:21.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:43:21.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:43:21.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:43:21.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:43:21.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:43:21.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:43:21.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:43:21.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:43:21.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:43:21.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:43:21.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:43:21.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:43:21.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:43:21.388 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:43:21.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:43:21.390 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:43:21.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:43:21.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:43:21.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:43:21.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:43:21.419 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:43:21.410+0000 7f5661016540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:43:21.423 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:43:21.414+0000 7f5661016540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:43:21.425 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:43:21.418+0000 7f5661016540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:43:21.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:43:21.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:43:21.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:43:21.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:43:21.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:43:21.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:43:21.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:43:21.704 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:43:21.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:43:21.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:43:21.967 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:43:21.958+0000 7f5661016540 -1 Falling back to public interface 2024-09-18T15:43:22.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:43:23.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:43:23.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:43:23.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:43:23.038 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:43:23.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:43:23.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:43:23.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:43:24.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:43:24.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:43:24.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:43:24.344 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:43:24.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:43:24.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:43:24.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:43:24.669 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:43:24.662+0000 7f5661016540 -1 osd.0 132 log_to_monitors true 2024-09-18T15:43:25.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:43:25.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:43:25.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:43:25.664 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:43:25.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:43:25.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:43:26.070 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 136 up_thru 136 down_at 133 last_clean_interval [107,132) [v2:127.0.0.1:6802/380187,v1:127.0.0.1:6803/380187] [v2:127.0.0.1:6804/380187,v1:127.0.0.1:6805/380187] exists,up 537f9958-5809-4e44-8060-20ee457d6502 2024-09-18T15:43:26.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:43:26.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:43:26.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:43:26.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:43:26.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:43:26.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:43:26.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:43:26.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:43:26.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:43:26.073 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:43:26.073 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:43:26.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:43:26.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:43:26.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:43:26.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:43:26.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:43:26.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:43:26.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:43:26.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:43:26.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:43:26.611 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:43:26.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:43:26.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:43:26.611 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:43:26.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=584115552258 2024-09-18T15:43:26.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 584115552258 2024-09-18T15:43:26.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-584115552258' 2024-09-18T15:43:26.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:43:26.786 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:43:26.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=558345748485 2024-09-18T15:43:26.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 558345748485 2024-09-18T15:43:26.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-584115552258 1-558345748485' 2024-09-18T15:43:26.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:26.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-584115552258 2024-09-18T15:43:26.924 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:26.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:43:26.927 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-584115552258 2024-09-18T15:43:26.928 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:26.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=584115552258 2024-09-18T15:43:26.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 584115552258' 2024-09-18T15:43:26.929 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 584115552258 2024-09-18T15:43:26.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:43:27.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 584115552258 -lt 584115552258 2024-09-18T15:43:27.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:27.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-558345748485 2024-09-18T15:43:27.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:27.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:43:27.244 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-558345748485 2024-09-18T15:43:27.244 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:27.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=558345748485 2024-09-18T15:43:27.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 558345748485' 2024-09-18T15:43:27.247 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 558345748485 2024-09-18T15:43:27.247 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:43:27.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 558345748485 -lt 558345748485 2024-09-18T15:43:27.577 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:43:27.577 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:43:27.577 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:43:27.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:43:27.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:43:27.942 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:43:27.942 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:43:27.942 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:43:27.942 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:43:27.943 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:43:27.943 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:43:28.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:43:28.270 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:43:28.271 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:43:28.271 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:43:28.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:43:28.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:43:28.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:43:28.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1125: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-09-18T15:43:28.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:43:28.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:43:28.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:43:28.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:43:28.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-09-18T15:43:28.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:43:28.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:43:28.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:43:28.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:43:28.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:43:28.639 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:43:28.639 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:43:28.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:43:28.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:43:28.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:43:28.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:43:28.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-09-18T15:43:28.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:43:28.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:43:28.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:43:28.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:43:28.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:43:28.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ14 rm-attr _ 2024-09-18T15:43:31.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:43:31.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:43:31.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:43:31.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:43:31.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:43:31.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:43:31.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:43:31.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:43:31.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:43:31.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:43:31.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:43:31.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:43:31.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:43:31.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:43:31.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:43:31.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:43:31.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:43:31.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:43:31.425 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:43:31.425 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:43:31.425 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:43:31.426 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:43:31.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:43:31.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:43:31.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:43:31.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:43:31.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:43:31.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:43:31.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:43:31.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:43:31.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:43:31.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:43:31.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:43:31.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:43:31.429 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:43:31.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:43:31.430 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:43:31.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:43:31.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:43:31.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:43:31.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:43:31.463 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:43:31.454+0000 7f55250e4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:43:31.466 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:43:31.458+0000 7f55250e4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:43:31.467 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:43:31.458+0000 7f55250e4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:43:31.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:43:31.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:43:31.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:43:31.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:43:31.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:43:31.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:43:31.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:43:31.742 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:43:31.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:43:31.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:43:32.007 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:43:31.998+0000 7f55250e4540 -1 Falling back to public interface 2024-09-18T15:43:32.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:43:33.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:43:33.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:43:33.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:43:33.079 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:43:33.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:43:33.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:43:33.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:43:34.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:43:34.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:43:34.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:43:34.396 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:43:34.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:43:34.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:43:34.479 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:43:34.470+0000 7f55250e4540 -1 osd.1 137 log_to_monitors true 2024-09-18T15:43:34.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:43:35.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:43:35.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:43:35.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:43:35.705 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:43:35.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:43:35.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:43:36.104 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 141 up_thru 141 down_at 138 last_clean_interval [130,137) [v2:127.0.0.1:6810/381465,v1:127.0.0.1:6811/381465] [v2:127.0.0.1:6812/381465,v1:127.0.0.1:6813/381465] exists,up 8c8956ed-6b0c-4a8b-8558-811b867c0d90 2024-09-18T15:43:36.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:43:36.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:43:36.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:43:36.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:43:36.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:43:36.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:43:36.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:43:36.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:43:36.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:43:36.106 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:43:36.106 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:43:36.107 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:43:36.107 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:43:36.107 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:43:36.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:43:36.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:43:36.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:43:36.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:43:36.334 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:43:36.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:43:36.643 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:43:36.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:43:36.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:43:36.644 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:43:36.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=584115552261 2024-09-18T15:43:36.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 584115552261 2024-09-18T15:43:36.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-584115552261' 2024-09-18T15:43:36.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:43:36.814 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:43:36.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=605590388738 2024-09-18T15:43:36.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 605590388738 2024-09-18T15:43:36.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-584115552261 1-605590388738' 2024-09-18T15:43:36.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:36.965 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-584115552261 2024-09-18T15:43:36.965 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:36.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:43:36.968 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-584115552261 2024-09-18T15:43:36.968 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:36.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=584115552261 2024-09-18T15:43:36.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 584115552261' 2024-09-18T15:43:36.971 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 584115552261 2024-09-18T15:43:36.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:43:37.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 584115552261 -lt 584115552261 2024-09-18T15:43:37.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:37.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-605590388738 2024-09-18T15:43:37.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:37.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:43:37.310 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-605590388738 2024-09-18T15:43:37.311 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:37.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=605590388738 2024-09-18T15:43:37.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 605590388738' 2024-09-18T15:43:37.313 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 605590388738 2024-09-18T15:43:37.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:43:37.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 605590388738 -lt 605590388738 2024-09-18T15:43:37.671 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:43:37.671 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:43:37.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:43:38.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:43:38.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:43:38.130 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:43:38.130 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:43:38.130 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:43:38.130 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:43:38.130 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:43:38.130 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:43:38.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:43:38.430 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:43:38.431 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:43:38.431 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:43:38.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:43:38.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:43:38.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:43:38.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1126: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val 2024-09-18T15:43:38.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:43:38.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-09-18T15:43:38.820 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: expr 15 % 2 2024-09-18T15:43:38.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: local osd=1 2024-09-18T15:43:38.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: case $i in 2024-09-18T15:43:38.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1130: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-09-18T15:43:38.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:43:38.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:43:38.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:43:38.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:43:38.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-09-18T15:43:38.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:43:38.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:43:38.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:43:38.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:43:38.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:43:38.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:43:38.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:43:38.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:43:38.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:43:38.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:43:39.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:43:39.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-09-18T15:43:39.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:43:39.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:43:39.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:43:39.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:43:39.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:43:39.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ15 rm-attr _ 2024-09-18T15:43:40.317 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-18T15:43:40.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:43:40.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:43:40.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:43:40.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:43:40.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:43:40.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:43:40.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:43:40.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:43:40.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:43:40.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:43:40.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:43:40.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:43:40.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:43:40.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:43:40.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:43:40.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:43:40.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:43:40.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:43:40.857 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:43:40.857 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:43:40.857 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:43:40.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:43:40.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:43:40.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:43:40.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:43:40.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:43:40.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:43:40.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:43:40.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:43:40.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:43:40.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:43:40.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:43:40.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:43:40.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:43:40.860 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:43:40.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:43:40.862 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:43:40.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:43:40.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:43:40.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:43:40.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:43:40.892 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:43:40.882+0000 7f45858c7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:43:40.895 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:43:40.886+0000 7f45858c7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:43:40.897 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:43:40.890+0000 7f45858c7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:43:41.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:43:41.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:43:41.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:43:41.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:43:41.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:43:41.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:43:41.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:43:41.171 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:43:41.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:43:41.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:43:41.439 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:43:41.430+0000 7f45858c7540 -1 Falling back to public interface 2024-09-18T15:43:41.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:43:42.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:43:42.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:43:42.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:43:42.476 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:43:42.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:43:42.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:43:42.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:43:43.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:43:43.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:43:43.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:43:43.792 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:43:43.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:43:43.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:43:43.927 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:43:43.918+0000 7f45858c7540 -1 osd.1 143 log_to_monitors true 2024-09-18T15:43:44.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:43:45.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:43:45.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:43:45.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:43:45.172 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:43:45.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:43:45.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:43:45.569 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 147 up_thru 147 down_at 144 last_clean_interval [141,143) [v2:127.0.0.1:6810/382745,v1:127.0.0.1:6811/382745] [v2:127.0.0.1:6812/382745,v1:127.0.0.1:6813/382745] exists,up 8c8956ed-6b0c-4a8b-8558-811b867c0d90 2024-09-18T15:43:45.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:43:45.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:43:45.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:43:45.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:43:45.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:43:45.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:43:45.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:43:45.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:43:45.571 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:43:45.572 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:43:45.572 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:43:45.572 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:43:45.572 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:43:45.572 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:43:45.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:43:45.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:43:45.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:43:45.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:43:45.801 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:43:46.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:43:46.114 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:43:46.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:43:46.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:43:46.115 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:43:46.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=584115552264 2024-09-18T15:43:46.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 584115552264 2024-09-18T15:43:46.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-584115552264' 2024-09-18T15:43:46.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:43:46.300 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:43:46.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=631360192514 2024-09-18T15:43:46.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 631360192514 2024-09-18T15:43:46.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-584115552264 1-631360192514' 2024-09-18T15:43:46.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:46.437 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-584115552264 2024-09-18T15:43:46.438 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:46.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:43:46.441 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-584115552264 2024-09-18T15:43:46.441 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:46.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=584115552264 2024-09-18T15:43:46.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 584115552264' 2024-09-18T15:43:46.444 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 584115552264 2024-09-18T15:43:46.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:43:46.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 584115552262 -lt 584115552264 2024-09-18T15:43:46.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:43:47.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:43:47.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:43:48.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 584115552264 -lt 584115552264 2024-09-18T15:43:48.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:48.095 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-631360192514 2024-09-18T15:43:48.095 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:48.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:43:48.099 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-631360192514 2024-09-18T15:43:48.099 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:48.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=631360192514 2024-09-18T15:43:48.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 631360192514' 2024-09-18T15:43:48.102 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 631360192514 2024-09-18T15:43:48.102 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:43:48.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 631360192514 -lt 631360192514 2024-09-18T15:43:48.449 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:43:48.449 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:43:48.450 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:43:48.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:43:48.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:43:48.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:43:48.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:43:48.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:43:48.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:43:48.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:43:48.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:43:49.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:43:49.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:43:49.206 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:43:49.206 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:43:49.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:43:49.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:43:49.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:43:49.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:43:49.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-09-18T15:43:49.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: expr 16 % 2 2024-09-18T15:43:49.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: local osd=0 2024-09-18T15:43:49.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: case $i in 2024-09-18T15:43:49.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1134: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-09-18T15:43:49.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:43:49.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:43:49.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:43:49.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:43:49.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-09-18T15:43:49.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:43:49.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:43:49.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:43:49.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:43:49.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:43:49.571 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:43:49.571 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:43:49.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:43:49.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:43:49.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:43:49.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:43:49.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-09-18T15:43:49.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:43:49.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:43:49.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:43:49.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:43:49.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:43:49.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ16 rm-attr snapset 2024-09-18T15:43:51.543 INFO:tasks.workunit.client.0.smithi046.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-09-18T15:43:52.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-18T15:43:52.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:43:52.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:43:52.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:43:52.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:43:52.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:43:52.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:43:52.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:43:52.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:43:52.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:43:52.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:43:52.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:43:52.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:43:52.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:43:52.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:43:52.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:43:52.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:43:52.095 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:43:52.097 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:43:52.097 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:43:52.097 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:43:52.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:43:52.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:43:52.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:43:52.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:43:52.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:43:52.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:43:52.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:43:52.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:43:52.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:43:52.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:43:52.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:43:52.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:43:52.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:43:52.098 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:43:52.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:43:52.099 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:43:52.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:43:52.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:43:52.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:43:52.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:43:52.126 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:43:52.118+0000 7f5eac42f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:43:52.130 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:43:52.122+0000 7f5eac42f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:43:52.132 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:43:52.122+0000 7f5eac42f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:43:52.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:43:52.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:43:52.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:43:52.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:43:52.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:43:52.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:43:52.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:43:52.414 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:43:52.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:43:52.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:43:52.671 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:43:52.662+0000 7f5eac42f540 -1 Falling back to public interface 2024-09-18T15:43:52.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:43:53.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:43:53.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:43:53.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:43:53.751 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:43:53.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:43:53.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:43:54.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:43:55.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:43:55.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:43:55.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:43:55.087 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:43:55.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:43:55.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:43:55.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:43:55.646 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:43:55.638+0000 7f5eac42f540 -1 osd.0 149 log_to_monitors true 2024-09-18T15:43:56.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:43:56.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:43:56.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:43:56.432 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:43:56.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:43:56.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:43:56.883 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 153 up_thru 153 down_at 150 last_clean_interval [136,149) [v2:127.0.0.1:6802/384048,v1:127.0.0.1:6803/384048] [v2:127.0.0.1:6804/384048,v1:127.0.0.1:6805/384048] exists,up 537f9958-5809-4e44-8060-20ee457d6502 2024-09-18T15:43:56.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:43:56.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:43:56.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:43:56.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:43:56.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:43:56.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:43:56.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:43:56.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:43:56.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:43:56.884 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:43:56.885 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:43:56.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:43:56.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:43:56.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:43:57.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:43:57.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:43:57.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:43:57.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:43:57.114 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:43:57.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:43:57.499 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:43:57.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:43:57.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:43:57.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:43:57.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=657129996290 2024-09-18T15:43:57.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 657129996290 2024-09-18T15:43:57.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-657129996290' 2024-09-18T15:43:57.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:43:57.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:43:57.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=631360192517 2024-09-18T15:43:57.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 631360192517 2024-09-18T15:43:57.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-657129996290 1-631360192517' 2024-09-18T15:43:57.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:57.837 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-657129996290 2024-09-18T15:43:57.838 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:57.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:43:57.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-657129996290 2024-09-18T15:43:57.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:57.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=657129996290 2024-09-18T15:43:57.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 657129996290' 2024-09-18T15:43:57.844 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 657129996290 2024-09-18T15:43:57.845 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:43:58.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 657129996290 2024-09-18T15:43:58.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:43:59.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:43:59.157 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:43:59.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 657129996290 -lt 657129996290 2024-09-18T15:43:59.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:59.492 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-631360192517 2024-09-18T15:43:59.492 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:59.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:43:59.496 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-631360192517 2024-09-18T15:43:59.496 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:59.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=631360192517 2024-09-18T15:43:59.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 631360192517' 2024-09-18T15:43:59.497 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 631360192517 2024-09-18T15:43:59.498 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:43:59.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 631360192517 -lt 631360192517 2024-09-18T15:43:59.836 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:43:59.837 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:43:59.837 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:44:00.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:44:00.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:44:00.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:44:00.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:44:00.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:44:00.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:44:00.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:44:00.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:44:00.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:44:00.596 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:44:00.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:44:00.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:44:00.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:44:00.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:44:00.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:44:00.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1135: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-09-18T15:44:00.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1136: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-09-18T15:44:00.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:44:00.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:44:00.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:44:00.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:44:00.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-09-18T15:44:00.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:44:00.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:44:00.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:44:00.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:44:00.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:44:00.963 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:44:00.963 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:44:00.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:44:00.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:44:00.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:44:02.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:44:02.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-09-18T15:44:02.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:44:02.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:44:02.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:44:02.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:44:02.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:44:02.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-09-18T15:44:03.092 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-18T15:44:03.092 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-18T15:44:03.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:44:03.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:44:03.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:44:03.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:44:03.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:44:03.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:44:03.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:44:03.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:44:03.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:44:03.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:44:03.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:44:03.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:44:03.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:44:03.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:44:03.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:44:03.629 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:44:03.629 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:44:03.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:44:03.630 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:44:03.630 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:44:03.630 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:44:03.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:44:03.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:44:03.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:44:03.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:44:03.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:44:03.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:44:03.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:44:03.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:44:03.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:44:03.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:44:03.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:44:03.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:44:03.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:44:03.633 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:44:03.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:44:03.634 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:44:03.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:44:03.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:44:03.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:44:03.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:44:03.666 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:03.658+0000 7efdeec55540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:44:03.669 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:03.662+0000 7efdeec55540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:44:03.671 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:03.662+0000 7efdeec55540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:44:03.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:44:03.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:44:03.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:44:03.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:44:03.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:44:03.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:44:03.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:44:03.950 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:44:03.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:44:03.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:44:04.203 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:04.194+0000 7efdeec55540 -1 Falling back to public interface 2024-09-18T15:44:04.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:44:05.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:44:05.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:44:05.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:44:05.255 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:44:05.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:44:05.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:44:05.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:44:06.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:44:06.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:44:06.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:44:06.563 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:44:06.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:44:06.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:44:06.735 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:06.725+0000 7efdeec55540 -1 osd.1 155 log_to_monitors true 2024-09-18T15:44:06.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:44:07.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:44:07.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:44:07.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:44:07.956 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:44:07.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:44:07.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:44:08.320 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 159 up_thru 159 down_at 156 last_clean_interval [147,155) [v2:127.0.0.1:6810/385351,v1:127.0.0.1:6811/385351] [v2:127.0.0.1:6812/385351,v1:127.0.0.1:6813/385351] exists,up 8c8956ed-6b0c-4a8b-8558-811b867c0d90 2024-09-18T15:44:08.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:44:08.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:44:08.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:44:08.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:44:08.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:44:08.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:44:08.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:44:08.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:44:08.321 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:44:08.322 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:44:08.322 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:44:08.322 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:44:08.322 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:44:08.323 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:44:08.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:44:08.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:44:08.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:44:08.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:44:08.545 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:44:08.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:44:08.857 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:44:08.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:44:08.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:44:08.858 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:44:09.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=657129996293 2024-09-18T15:44:09.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 657129996293 2024-09-18T15:44:09.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-657129996293' 2024-09-18T15:44:09.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:44:09.025 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:44:09.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=682899800066 2024-09-18T15:44:09.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 682899800066 2024-09-18T15:44:09.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-657129996293 1-682899800066' 2024-09-18T15:44:09.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:44:09.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-657129996293 2024-09-18T15:44:09.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:44:09.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:44:09.179 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-657129996293 2024-09-18T15:44:09.180 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:44:09.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=657129996293 2024-09-18T15:44:09.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 657129996293' 2024-09-18T15:44:09.181 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 657129996293 2024-09-18T15:44:09.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:44:09.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 657129996293 -lt 657129996293 2024-09-18T15:44:09.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:44:09.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-682899800066 2024-09-18T15:44:09.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:44:09.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:44:09.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-682899800066 2024-09-18T15:44:09.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:44:09.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=682899800066 2024-09-18T15:44:09.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 682899800066' 2024-09-18T15:44:09.500 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 682899800066 2024-09-18T15:44:09.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:44:09.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 682899800066 2024-09-18T15:44:09.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:44:10.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:44:10.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:44:11.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 682899800066 2024-09-18T15:44:11.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:44:12.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-18T15:44:12.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:44:12.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 682899800066 -lt 682899800066 2024-09-18T15:44:12.470 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:44:12.471 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:44:12.471 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:44:12.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:44:12.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:44:12.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:44:12.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:44:12.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:44:12.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:44:12.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:44:12.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:44:13.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:44:13.221 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:44:13.221 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:44:13.222 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:44:13.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:44:13.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:44:13.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:44:13.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:44:13.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-09-18T15:44:13.586 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: expr 17 % 2 2024-09-18T15:44:13.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: local osd=1 2024-09-18T15:44:13.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: case $i in 2024-09-18T15:44:13.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1141: TEST_corrupt_scrub_replicated: local payload=ROBJ17 2024-09-18T15:44:13.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1142: TEST_corrupt_scrub_replicated: echo ROBJ17 2024-09-18T15:44:13.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1143: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-18T15:44:13.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:44:13.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:44:13.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:44:13.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:44:13.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-18T15:44:13.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:44:13.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:44:13.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:44:13.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:44:13.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:44:13.590 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:44:13.590 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:44:13.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:44:13.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:44:13.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:44:13.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:44:13.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-18T15:44:13.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:44:13.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:44:13.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:44:13.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:44:13.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:44:13.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-18T15:44:15.494 INFO:tasks.workunit.client.0.smithi046.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-09-18T15:44:16.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-18T15:44:16.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:44:16.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:44:16.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:44:16.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:44:16.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:44:16.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:44:16.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:44:16.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:44:16.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:44:16.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:44:16.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:44:16.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:44:16.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:44:16.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:44:16.129 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:44:16.129 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:44:16.129 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:44:16.130 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:44:16.130 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:44:16.130 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:44:16.130 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:44:16.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:44:16.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:44:16.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:44:16.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:44:16.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:44:16.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:44:16.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:44:16.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:44:16.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:44:16.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:44:16.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:44:16.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:44:16.133 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:44:16.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:44:16.134 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:44:16.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:44:16.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:44:16.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:44:16.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:44:16.165 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:16.157+0000 7f2466dd2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:44:16.168 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:16.161+0000 7f2466dd2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:44:16.172 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:16.161+0000 7f2466dd2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:44:16.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:44:16.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:44:16.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:44:16.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:44:16.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:44:16.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:44:16.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:44:16.444 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:44:16.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:44:16.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:44:16.711 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:16.701+0000 7f2466dd2540 -1 Falling back to public interface 2024-09-18T15:44:16.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:44:17.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:44:17.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:44:17.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:44:17.802 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:44:17.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:44:17.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:44:18.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:44:19.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:44:19.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:44:19.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:44:19.119 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:44:19.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:44:19.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:44:19.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:44:19.753 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:19.745+0000 7f2466dd2540 -1 osd.0 161 log_to_monitors true 2024-09-18T15:44:20.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:44:20.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:44:20.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:44:20.427 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:44:20.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:44:20.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:44:20.970 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 165 up_thru 159 down_at 162 last_clean_interval [153,161) [v2:127.0.0.1:6802/386678,v1:127.0.0.1:6803/386678] [v2:127.0.0.1:6804/386678,v1:127.0.0.1:6805/386678] exists,up 537f9958-5809-4e44-8060-20ee457d6502 2024-09-18T15:44:20.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:44:20.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:44:20.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:44:20.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:44:20.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:44:20.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:44:20.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:44:20.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:44:20.971 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:44:20.972 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:44:20.973 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:44:20.973 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:44:20.973 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:44:20.973 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:44:21.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:44:21.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:44:21.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:44:21.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:44:21.209 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:44:21.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:44:21.601 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:44:21.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:44:21.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:44:21.601 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:44:21.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=708669603842 2024-09-18T15:44:21.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 708669603842 2024-09-18T15:44:21.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-708669603842' 2024-09-18T15:44:21.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:44:21.747 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:44:21.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=682899800069 2024-09-18T15:44:21.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 682899800069 2024-09-18T15:44:21.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-708669603842 1-682899800069' 2024-09-18T15:44:21.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:44:21.916 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-708669603842 2024-09-18T15:44:21.916 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:44:21.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:44:21.919 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-708669603842 2024-09-18T15:44:21.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:44:21.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=708669603842 2024-09-18T15:44:21.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 708669603842' 2024-09-18T15:44:21.922 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 708669603842 2024-09-18T15:44:21.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:44:22.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 708669603842 2024-09-18T15:44:22.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:44:23.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:44:23.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:44:23.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 708669603842 -lt 708669603842 2024-09-18T15:44:23.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:44:23.562 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-682899800069 2024-09-18T15:44:23.562 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:44:23.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:44:23.566 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-682899800069 2024-09-18T15:44:23.566 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:44:23.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=682899800069 2024-09-18T15:44:23.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 682899800069' 2024-09-18T15:44:23.568 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 682899800069 2024-09-18T15:44:23.569 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:44:23.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 682899800070 -lt 682899800069 2024-09-18T15:44:23.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:44:23.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:44:23.912 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:44:24.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:44:24.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:44:24.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:44:24.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:44:24.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:44:24.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:44:24.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:44:24.320 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:44:24.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:44:24.652 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:44:24.653 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:44:24.653 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:44:25.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:44:25.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:44:25.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:44:25.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1144: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-18T15:44:25.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:44:25.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:44:25.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:44:25.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:44:25.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-18T15:44:25.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:44:25.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:44:25.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:44:25.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:44:25.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:44:25.024 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:44:25.024 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:44:25.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:44:25.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:44:25.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:44:25.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:44:25.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-18T15:44:25.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:44:25.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:44:25.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:44:25.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:44:25.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:44:25.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-18T15:44:26.435 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-18T15:44:26.435 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-18T15:44:26.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:44:26.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:44:26.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:44:26.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:44:26.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:44:26.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:44:26.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:44:26.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:44:26.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:44:26.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:44:26.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:44:26.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:44:26.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:44:26.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:44:26.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:44:26.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:44:26.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:44:26.998 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:44:26.998 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:44:26.998 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:44:26.998 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:44:26.998 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:44:26.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:44:26.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:44:26.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:44:26.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:44:26.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:44:26.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:44:27.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:44:27.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:44:27.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:44:27.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:44:27.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:44:27.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:44:27.001 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:44:27.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:44:27.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:44:27.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:44:27.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:44:27.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:44:27.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:44:27.033 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:27.025+0000 7f31dfaf9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:44:27.036 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:27.029+0000 7f31dfaf9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:44:27.040 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:27.029+0000 7f31dfaf9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:44:27.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:44:27.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:44:27.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:44:27.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:44:27.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:44:27.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:44:27.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:44:27.313 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:44:27.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:44:27.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:44:27.570 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:27.561+0000 7f31dfaf9540 -1 Falling back to public interface 2024-09-18T15:44:27.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:44:28.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:44:28.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:44:28.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:44:28.652 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:44:28.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:44:28.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:44:28.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:44:29.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:44:29.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:44:29.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:44:29.966 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:44:29.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:44:29.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:44:30.120 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:30.109+0000 7f31dfaf9540 -1 osd.1 167 log_to_monitors true 2024-09-18T15:44:30.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:44:31.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:44:31.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:44:31.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:44:31.279 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:44:31.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:44:31.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:44:31.621 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 171 up_thru 171 down_at 168 last_clean_interval [159,167) [v2:127.0.0.1:6810/387980,v1:127.0.0.1:6811/387980] [v2:127.0.0.1:6812/387980,v1:127.0.0.1:6813/387980] exists,up 8c8956ed-6b0c-4a8b-8558-811b867c0d90 2024-09-18T15:44:31.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:44:31.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:44:31.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:44:31.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:44:31.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:44:31.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:44:31.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:44:31.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:44:31.622 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:44:31.623 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:44:31.623 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:44:31.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:44:31.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:44:31.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:44:31.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:44:31.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:44:31.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:44:31.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:44:31.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:44:32.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:44:32.166 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:44:32.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:44:32.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:44:32.167 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:44:32.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=708669603845 2024-09-18T15:44:32.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 708669603845 2024-09-18T15:44:32.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-708669603845' 2024-09-18T15:44:32.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:44:32.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:44:32.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=734439407618 2024-09-18T15:44:32.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 734439407618 2024-09-18T15:44:32.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-708669603845 1-734439407618' 2024-09-18T15:44:32.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:44:32.473 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-708669603845 2024-09-18T15:44:32.473 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:44:32.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:44:32.477 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-708669603845 2024-09-18T15:44:32.477 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:44:32.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=708669603845 2024-09-18T15:44:32.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 708669603845' 2024-09-18T15:44:32.480 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 708669603845 2024-09-18T15:44:32.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:44:32.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 708669603844 -lt 708669603845 2024-09-18T15:44:32.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:44:33.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:44:33.790 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:44:34.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 708669603845 -lt 708669603845 2024-09-18T15:44:34.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:44:34.108 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-734439407618 2024-09-18T15:44:34.108 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:44:34.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:44:34.112 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-734439407618 2024-09-18T15:44:34.112 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:44:34.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=734439407618 2024-09-18T15:44:34.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 734439407618' 2024-09-18T15:44:34.114 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 734439407618 2024-09-18T15:44:34.115 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:44:34.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 734439407618 -lt 734439407618 2024-09-18T15:44:34.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:44:34.462 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:44:34.462 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:44:34.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:44:34.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:44:34.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:44:34.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:44:34.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:44:34.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:44:34.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:44:34.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:44:35.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:44:35.225 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:44:35.226 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:44:35.226 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:44:35.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:44:35.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:44:35.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:44:35.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:44:35.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-09-18T15:44:35.588 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: expr 18 % 2 2024-09-18T15:44:35.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: local osd=0 2024-09-18T15:44:35.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: case $i in 2024-09-18T15:44:35.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1149: TEST_corrupt_scrub_replicated: local payload=ROBJ18 2024-09-18T15:44:35.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1150: TEST_corrupt_scrub_replicated: echo ROBJ18 2024-09-18T15:44:35.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1151: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-18T15:44:35.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:44:35.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:44:35.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:44:35.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:44:35.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-18T15:44:35.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:44:35.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:44:35.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:44:35.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:44:35.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:44:35.592 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:44:35.592 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:44:35.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:44:35.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:44:35.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:44:35.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:44:35.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-18T15:44:35.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:44:35.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:44:35.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:44:35.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:44:35.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:44:35.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-18T15:44:37.607 INFO:tasks.workunit.client.0.smithi046.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-09-18T15:44:38.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-18T15:44:38.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:44:38.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:44:38.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:44:38.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:44:38.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:44:38.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:44:38.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:44:38.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:44:38.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:44:38.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:44:38.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:44:38.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:44:38.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:44:38.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:44:38.237 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:44:38.237 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:44:38.237 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:44:38.237 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:44:38.238 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:44:38.238 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:44:38.239 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:44:38.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:44:38.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:44:38.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:44:38.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:44:38.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:44:38.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:44:38.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:44:38.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:44:38.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:44:38.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:44:38.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:44:38.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:44:38.240 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:44:38.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:44:38.242 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:44:38.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:44:38.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:44:38.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:44:38.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:44:38.272 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:38.261+0000 7f118ebd5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:44:38.274 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:38.265+0000 7f118ebd5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:44:38.276 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:38.269+0000 7f118ebd5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:44:38.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:44:38.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:44:38.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:44:38.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:44:38.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:44:38.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:44:38.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:44:38.554 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:44:38.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:44:38.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:44:38.811 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:38.801+0000 7f118ebd5540 -1 Falling back to public interface 2024-09-18T15:44:38.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:44:39.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:44:39.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:44:39.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:44:39.916 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:44:39.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:44:39.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:44:40.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:44:41.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:44:41.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:44:41.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:44:41.239 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:44:41.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:44:41.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:44:41.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:44:41.785 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:41.777+0000 7f118ebd5540 -1 osd.0 173 log_to_monitors true 2024-09-18T15:44:42.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:44:42.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:44:42.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:44:42.599 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:44:42.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:44:42.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:44:43.181 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 177 up_thru 177 down_at 174 last_clean_interval [165,173) [v2:127.0.0.1:6802/389283,v1:127.0.0.1:6803/389283] [v2:127.0.0.1:6804/389283,v1:127.0.0.1:6805/389283] exists,up 537f9958-5809-4e44-8060-20ee457d6502 2024-09-18T15:44:43.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:44:43.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:44:43.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:44:43.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:44:43.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:44:43.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:44:43.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:44:43.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:44:43.183 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:44:43.184 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:44:43.184 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:44:43.185 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:44:43.185 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:44:43.185 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:44:43.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:44:43.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:44:43.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:44:43.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:44:43.418 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:44:43.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:44:43.730 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:44:43.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:44:43.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:44:43.730 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:44:43.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=760209211394 2024-09-18T15:44:43.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 760209211394 2024-09-18T15:44:43.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-760209211394' 2024-09-18T15:44:43.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:44:43.916 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:44:44.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=734439407621 2024-09-18T15:44:44.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 734439407621 2024-09-18T15:44:44.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-760209211394 1-734439407621' 2024-09-18T15:44:44.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:44:44.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-760209211394 2024-09-18T15:44:44.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:44:44.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:44:44.063 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-760209211394 2024-09-18T15:44:44.063 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:44:44.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=760209211394 2024-09-18T15:44:44.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 760209211394' 2024-09-18T15:44:44.065 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 760209211394 2024-09-18T15:44:44.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:44:44.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 760209211394 2024-09-18T15:44:44.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:44:45.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:44:45.381 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:44:45.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 760209211394 -lt 760209211394 2024-09-18T15:44:45.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:44:45.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-734439407621 2024-09-18T15:44:45.694 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:44:45.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:44:45.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-734439407621 2024-09-18T15:44:45.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:44:45.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=734439407621 2024-09-18T15:44:45.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 734439407621' 2024-09-18T15:44:45.700 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 734439407621 2024-09-18T15:44:45.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:44:46.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 734439407621 -lt 734439407621 2024-09-18T15:44:46.045 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:44:46.045 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:44:46.046 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:44:46.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:44:46.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:44:46.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:44:46.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:44:46.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:44:46.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:44:46.460 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:44:46.460 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:44:46.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:44:46.790 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:44:46.790 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:44:46.791 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:44:47.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:44:47.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:44:47.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:44:47.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1152: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-18T15:44:47.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:44:47.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:44:47.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:44:47.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:44:47.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-18T15:44:47.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:44:47.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:44:47.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:44:47.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:44:47.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:44:47.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:44:47.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:44:47.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:44:47.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:44:47.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:44:47.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:44:47.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-18T15:44:47.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:44:47.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:44:47.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:44:47.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:44:47.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:44:47.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-18T15:44:48.573 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-18T15:44:48.573 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-18T15:44:49.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:44:49.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:44:49.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:44:49.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:44:49.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:44:49.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:44:49.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:44:49.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:44:49.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:44:49.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:44:49.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:44:49.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:44:49.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:44:49.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:44:49.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:44:49.129 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:44:49.129 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:44:49.129 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:44:49.129 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:44:49.129 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:44:49.129 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:44:49.131 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:44:49.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:44:49.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:44:49.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:44:49.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:44:49.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:44:49.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:44:49.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:44:49.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:44:49.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:44:49.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:44:49.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:44:49.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:44:49.133 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:44:49.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:44:49.134 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:44:49.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:44:49.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:44:49.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:44:49.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:44:49.165 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:49.157+0000 7fc682d36540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:44:49.168 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:49.157+0000 7fc682d36540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:44:49.169 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:49.161+0000 7fc682d36540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:44:49.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:44:49.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:44:49.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:44:49.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:44:49.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:44:49.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:44:49.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:44:49.445 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:44:49.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:44:49.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:44:49.711 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:49.701+0000 7fc682d36540 -1 Falling back to public interface 2024-09-18T15:44:49.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:44:50.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:44:50.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:44:50.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:44:50.848 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:44:50.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:44:50.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:44:51.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:44:52.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:44:52.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:44:52.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:44:52.174 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:44:52.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:44:52.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:44:52.244 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:52.233+0000 7fc682d36540 -1 osd.1 179 log_to_monitors true 2024-09-18T15:44:52.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:44:53.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:44:53.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:44:53.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:44:53.499 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:44:53.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:44:53.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:44:53.857 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 183 up_thru 183 down_at 180 last_clean_interval [171,179) [v2:127.0.0.1:6810/390585,v1:127.0.0.1:6811/390585] [v2:127.0.0.1:6812/390585,v1:127.0.0.1:6813/390585] exists,up 8c8956ed-6b0c-4a8b-8558-811b867c0d90 2024-09-18T15:44:53.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:44:53.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:44:53.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:44:53.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:44:53.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:44:53.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:44:53.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:44:53.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:44:53.858 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:44:53.859 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:44:53.859 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:44:53.860 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:44:53.860 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:44:53.860 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:44:54.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:44:54.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:44:54.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:44:54.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:44:54.083 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:44:54.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:44:54.402 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:44:54.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:44:54.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:44:54.402 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:44:54.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=760209211397 2024-09-18T15:44:54.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 760209211397 2024-09-18T15:44:54.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-760209211397' 2024-09-18T15:44:54.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:44:54.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:44:54.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=785979015170 2024-09-18T15:44:54.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 785979015170 2024-09-18T15:44:54.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-760209211397 1-785979015170' 2024-09-18T15:44:54.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:44:54.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-760209211397 2024-09-18T15:44:54.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:44:54.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:44:54.725 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-760209211397 2024-09-18T15:44:54.726 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:44:54.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=760209211397 2024-09-18T15:44:54.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 760209211397' 2024-09-18T15:44:54.728 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 760209211397 2024-09-18T15:44:54.729 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:44:55.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 760209211396 -lt 760209211397 2024-09-18T15:44:55.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:44:56.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:44:56.033 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:44:56.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 760209211397 -lt 760209211397 2024-09-18T15:44:56.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:44:56.354 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-785979015170 2024-09-18T15:44:56.354 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:44:56.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:44:56.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-785979015170 2024-09-18T15:44:56.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:44:56.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=785979015170 2024-09-18T15:44:56.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 785979015170' 2024-09-18T15:44:56.360 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 785979015170 2024-09-18T15:44:56.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:44:56.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 785979015170 -lt 785979015170 2024-09-18T15:44:56.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:44:56.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:44:56.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:44:57.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:44:57.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:44:57.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:44:57.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:44:57.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:44:57.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:44:57.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:44:57.106 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:44:57.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:44:57.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:44:57.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:44:57.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:44:57.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:44:57.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:44:57.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:44:57.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1154: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-09-18T15:44:57.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:44:57.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:44:57.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:44:57.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:44:57.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-09-18T15:44:57.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:44:57.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:44:57.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:44:57.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:44:57.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:44:57.768 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:44:57.768 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:44:57.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:44:57.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:44:57.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:44:58.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:44:58.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-09-18T15:44:58.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:44:58.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:44:58.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:44:58.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:44:58.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:44:58.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 corrupt-info 2024-09-18T15:44:59.340 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-18T15:44:59.340 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-18T15:44:59.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:44:59.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:44:59.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:44:59.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:44:59.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:44:59.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:44:59.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:44:59.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:44:59.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:44:59.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:44:59.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:44:59.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:44:59.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:44:59.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:44:59.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:44:59.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:44:59.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:44:59.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:44:59.889 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:44:59.889 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:44:59.890 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:44:59.891 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:44:59.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:44:59.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:44:59.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:44:59.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:44:59.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:44:59.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:44:59.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:44:59.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:44:59.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:44:59.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:44:59.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:44:59.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:44:59.893 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:44:59.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:44:59.895 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:44:59.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:44:59.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:44:59.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:44:59.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:44:59.924 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:59.913+0000 7f26062a5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:44:59.927 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:59.917+0000 7f26062a5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:44:59.929 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:44:59.921+0000 7f26062a5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:45:00.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:45:00.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:45:00.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:45:00.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:45:00.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:45:00.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:00.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:45:00.204 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:45:00.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:00.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:45:00.467 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:45:00.457+0000 7f26062a5540 -1 Falling back to public interface 2024-09-18T15:45:00.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:01.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:01.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:01.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:45:01.554 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:45:01.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:01.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:45:01.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:02.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:02.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:02.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:45:02.911 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:45:02.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:02.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:45:03.132 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:45:03.121+0000 7f26062a5540 -1 osd.1 185 log_to_monitors true 2024-09-18T15:45:03.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:04.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:04.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:04.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:45:04.231 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:45:04.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:04.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:45:04.675 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 189 up_thru 189 down_at 186 last_clean_interval [183,185) [v2:127.0.0.1:6810/391887,v1:127.0.0.1:6811/391887] [v2:127.0.0.1:6812/391887,v1:127.0.0.1:6813/391887] exists,up 8c8956ed-6b0c-4a8b-8558-811b867c0d90 2024-09-18T15:45:04.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:45:04.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:45:04.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:45:04.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:45:04.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:45:04.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:45:04.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:45:04.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:45:04.676 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:45:04.677 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:45:04.677 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:45:04.677 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:45:04.678 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:45:04.678 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:45:04.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:45:04.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:45:04.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:45:04.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:45:04.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:45:05.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:45:05.231 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:45:05.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:45:05.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:45:05.231 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:45:05.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=760209211400 2024-09-18T15:45:05.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 760209211400 2024-09-18T15:45:05.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-760209211400' 2024-09-18T15:45:05.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:45:05.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:45:05.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=811748818946 2024-09-18T15:45:05.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 811748818946 2024-09-18T15:45:05.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-760209211400 1-811748818946' 2024-09-18T15:45:05.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:45:05.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:45:05.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-760209211400 2024-09-18T15:45:05.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:45:05.515 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-760209211400 2024-09-18T15:45:05.516 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:45:05.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=760209211400 2024-09-18T15:45:05.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 760209211400' 2024-09-18T15:45:05.518 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 760209211400 2024-09-18T15:45:05.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:45:05.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 760209211399 -lt 760209211400 2024-09-18T15:45:05.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:45:06.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:45:06.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:45:07.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 760209211399 -lt 760209211400 2024-09-18T15:45:07.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:45:08.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-18T15:45:08.170 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:45:08.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 760209211401 -lt 760209211400 2024-09-18T15:45:08.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:45:08.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-811748818946 2024-09-18T15:45:08.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:45:08.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:45:08.485 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-811748818946 2024-09-18T15:45:08.486 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:45:08.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=811748818946 2024-09-18T15:45:08.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 811748818946' 2024-09-18T15:45:08.487 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 811748818946 2024-09-18T15:45:08.488 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:45:08.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 811748818946 -lt 811748818946 2024-09-18T15:45:08.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:45:08.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:45:08.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:45:09.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:45:09.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:45:09.231 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:45:09.231 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:45:09.231 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:45:09.231 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:45:09.231 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:45:09.231 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:45:09.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:45:09.590 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:45:09.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:45:09.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:45:09.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:45:09.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:45:09.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:45:09.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-18T15:45:09.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-09-18T15:45:09.956 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: expr 19 % 2 2024-09-18T15:45:09.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: local osd=1 2024-09-18T15:45:09.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: case $i in 2024-09-18T15:45:09.959 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1163: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-09-18T15:45:09.959 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-09-18T15:45:09.959 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-09-18T15:45:09.960 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-09-18T15:45:09.960 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:45:10.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1163: TEST_corrupt_scrub_replicated: local pg=3.0 2024-09-18T15:45:10.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1165: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=1048576 2024-09-18T15:45:10.415 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0: {} 2024-09-18T15:45:10.434 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1: {} 2024-09-18T15:45:10.435 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-18T15:45:10.435 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-18T15:45:10.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1167: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-09-18T15:45:10.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:45:10.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:45:10.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:45:10.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:45:10.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=csr_pool 2024-09-18T15:45:10.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:45:10.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=ROBJ11 2024-09-18T15:45:10.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:45:10.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-scrub-repair 2024-09-18T15:45:10.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:45:10.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:45:10.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:45:10.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T15:45:10.447 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds csr_pool ROBJ11 2024-09-18T15:45:10.447 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-09-18T15:45:10.447 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ11 2024-09-18T15:45:10.447 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-09-18T15:45:10.447 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:45:10.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-09-18T15:45:10.749 INFO:tasks.workunit.client.0.smithi046.stderr:0' 2024-09-18T15:45:10.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-09-18T15:45:10.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:45:10.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-18T15:45:10.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:45:10.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:45:10.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-scrub-repair/1/type 2024-09-18T15:45:10.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:45:10.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-18T15:45:10.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:45:10.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T15:45:10.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:45:10.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:45:10.754 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:45:10.755 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T15:45:10.755 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:45:10.755 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:45:10.756 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:45:10.756 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:45:10.756 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:45:10.757 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-osd.1.asok 2024-09-18T15:45:10.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.137209/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:45:10.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:45:10.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:45:10.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:45:10.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-18T15:45:10.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:45:10.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:45:10.850 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:45:10.850 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:45:10.851 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:45:10.851 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-osd.1.asok 2024-09-18T15:45:10.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:45:10.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.137209/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-09-18T15:45:10.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1168: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-09-18T15:45:10.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:45:10.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:45:10.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=mdata 2024-09-18T15:45:10.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:45:10.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=csr_pool 2024-09-18T15:45:10.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:45:10.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=ROBJ12 2024-09-18T15:45:10.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:45:10.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-scrub-repair 2024-09-18T15:45:10.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:45:10.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-18T15:45:10.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:45:10.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T15:45:10.939 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds csr_pool ROBJ12 2024-09-18T15:45:10.939 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-09-18T15:45:10.939 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ12 2024-09-18T15:45:10.939 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-09-18T15:45:10.940 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:45:11.252 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-09-18T15:45:11.252 INFO:tasks.workunit.client.0.smithi046.stderr:0' 2024-09-18T15:45:11.252 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-09-18T15:45:11.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:45:11.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=0 2024-09-18T15:45:11.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:45:11.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:45:11.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-scrub-repair/0/type 2024-09-18T15:45:11.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:45:11.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-18T15:45:11.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:45:11.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-09-18T15:45:11.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:45:11.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:45:11.257 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:45:11.258 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-09-18T15:45:11.258 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T15:45:11.258 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T15:45:11.259 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:45:11.259 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:45:11.259 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:45:11.260 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-osd.0.asok 2024-09-18T15:45:11.260 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.137209/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:45:11.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:45:11.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:45:11.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:45:11.354 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.0 2024-09-18T15:45:11.354 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T15:45:11.354 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T15:45:11.355 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:45:11.355 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:45:11.355 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:45:11.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-osd.0.asok 2024-09-18T15:45:11.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:45:11.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.137209/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-09-18T15:45:11.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1169: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-09-18T15:45:11.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:45:11.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:45:11.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=mdata 2024-09-18T15:45:11.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:45:11.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=csr_pool 2024-09-18T15:45:11.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:45:11.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=ROBJ13 2024-09-18T15:45:11.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:45:11.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-scrub-repair 2024-09-18T15:45:11.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:45:11.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-18T15:45:11.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:45:11.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T15:45:11.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds csr_pool ROBJ13 2024-09-18T15:45:11.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-09-18T15:45:11.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-09-18T15:45:11.453 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-09-18T15:45:11.453 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:45:11.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-09-18T15:45:11.749 INFO:tasks.workunit.client.0.smithi046.stderr:0' 2024-09-18T15:45:11.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-09-18T15:45:11.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:45:11.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=0 2024-09-18T15:45:11.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:45:11.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:45:11.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-scrub-repair/0/type 2024-09-18T15:45:11.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:45:11.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-18T15:45:11.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:45:11.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-09-18T15:45:11.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:45:11.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:45:11.755 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:45:11.755 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-09-18T15:45:11.755 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T15:45:11.755 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T15:45:11.756 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:45:11.756 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:45:11.756 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:45:11.756 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-osd.0.asok 2024-09-18T15:45:11.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.137209/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:45:11.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:45:11.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:45:11.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:45:11.855 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.0 2024-09-18T15:45:11.855 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T15:45:11.855 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T15:45:11.856 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:45:11.856 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:45:11.856 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:45:11.856 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-osd.0.asok 2024-09-18T15:45:11.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:45:11.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.137209/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-09-18T15:45:11.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1170: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-09-18T15:45:11.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:45:11.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:45:11.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:45:11.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:45:11.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=csr_pool 2024-09-18T15:45:11.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:45:11.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=ROBJ13 2024-09-18T15:45:11.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:45:11.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-scrub-repair 2024-09-18T15:45:11.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:45:11.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:45:11.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:45:11.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T15:45:11.952 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds csr_pool ROBJ13 2024-09-18T15:45:11.952 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-09-18T15:45:11.952 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-09-18T15:45:11.952 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-09-18T15:45:11.953 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:45:12.252 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-09-18T15:45:12.253 INFO:tasks.workunit.client.0.smithi046.stderr:0' 2024-09-18T15:45:12.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-09-18T15:45:12.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:45:12.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-18T15:45:12.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:45:12.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:45:12.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-scrub-repair/1/type 2024-09-18T15:45:12.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:45:12.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-18T15:45:12.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:45:12.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T15:45:12.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:45:12.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:45:12.257 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:45:12.257 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T15:45:12.257 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:45:12.257 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:45:12.258 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:45:12.258 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:45:12.258 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:45:12.258 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-osd.1.asok 2024-09-18T15:45:12.258 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.137209/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:45:12.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:45:12.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:45:12.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:45:12.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-18T15:45:12.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:45:12.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:45:12.355 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:45:12.356 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:45:12.356 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:45:12.356 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-osd.1.asok 2024-09-18T15:45:12.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:45:12.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.137209/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-09-18T15:45:12.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1172: TEST_corrupt_scrub_replicated: pg_scrub 3.0 2024-09-18T15:45:12.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: pg_scrub: local pgid=3.0 2024-09-18T15:45:12.456 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: pg_scrub: get_last_scrub_stamp 3.0 2024-09-18T15:45:12.456 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:45:12.456 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:45:12.456 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:45:12.457 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:45:12.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: pg_scrub: local last_scrub=2024-09-18T15:40:57.142428+0000 2024-09-18T15:45:12.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_scrub: ceph pg scrub 3.0 2024-09-18T15:45:13.063 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 3.0 on osd.1 to scrub 2024-09-18T15:45:13.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_scrub: wait_for_scrub 3.0 2024-09-18T15:40:57.142428+0000 2024-09-18T15:45:13.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=3.0 2024-09-18T15:45:13.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:40:57.142428+0000 2024-09-18T15:45:13.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:45:13.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:45:13.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:45:13.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:45:13.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:45:13.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:45:13.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:45:13.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:45:13.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:40:57.142428+0000 '>' 2024-09-18T15:40:57.142428+0000 2024-09-18T15:45:13.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:45:14.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:45:14.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:45:14.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:45:14.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:45:14.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:45:14.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:45:14.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:45:14.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:40:57.142428+0000 '>' 2024-09-18T15:40:57.142428+0000 2024-09-18T15:45:14.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:45:15.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:45:15.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:45:15.707 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:45:15.707 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:45:15.707 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:45:15.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:45:15.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:45:16.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:40:57.142428+0000 '>' 2024-09-18T15:40:57.142428+0000 2024-09-18T15:45:16.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:45:17.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:45:17.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:45:17.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:45:17.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:45:17.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:45:17.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:45:17.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:45:17.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:40:57.142428+0000 '>' 2024-09-18T15:40:57.142428+0000 2024-09-18T15:45:17.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:45:18.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:45:18.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:45:18.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:45:18.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:45:18.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:45:18.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:45:18.362 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:45:18.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:40:57.142428+0000 '>' 2024-09-18T15:40:57.142428+0000 2024-09-18T15:45:18.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:45:19.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:45:19.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:45:19.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:45:19.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:45:19.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:45:19.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:45:19.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:45:19.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:45:16.965439+0000 '>' 2024-09-18T15:40:57.142428+0000 2024-09-18T15:45:19.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:45:19.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1174: TEST_corrupt_scrub_replicated: ERRORS=0 2024-09-18T15:45:19.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1175: TEST_corrupt_scrub_replicated: declare -a s_err_strings 2024-09-18T15:45:19.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1176: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-09-18T15:45:19.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1177: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' 2024-09-18T15:45:19.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1178: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-09-18T15:45:19.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1179: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-09-18T15:45:19.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1180: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-09-18T15:45:19.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1181: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-09-18T15:45:19.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-09-18T15:45:19.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1183: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-09-18T15:45:19.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-09-18T15:45:19.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1185: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-09-18T15:45:19.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1186: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-09-18T15:45:19.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1187: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-09-18T15:45:19.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1188: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-09-18T15:45:19.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1189: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-09-18T15:45:19.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1190: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' 2024-09-18T15:45:19.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1191: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-09-18T15:45:19.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1192: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-09-18T15:45:19.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' 2024-09-18T15:45:19.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1194: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-09-18T15:45:19.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' 2024-09-18T15:45:19.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1196: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' 2024-09-18T15:45:19.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' 2024-09-18T15:45:19.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:45:19.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-09-18T15:45:19.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:45:19.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-09-18T15:45:20.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:45:20.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-09-18T15:45:20.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:45:20.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-09-18T15:45:20.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:45:20.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-09-18T15:45:20.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:45:20.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-09-18T15:45:20.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:45:20.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-09-18T15:45:20.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:45:20.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-09-18T15:45:20.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:45:20.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-09-18T15:45:20.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:45:20.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-09-18T15:45:20.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:45:20.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-09-18T15:45:20.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:45:20.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-09-18T15:45:20.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:45:20.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-09-18T15:45:20.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:45:20.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-09-18T15:45:20.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:45:20.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-09-18T15:45:20.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:45:20.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-09-18T15:45:20.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:45:20.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-09-18T15:45:20.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:45:20.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' td/osd-scrub-repair/osd.1.log 2024-09-18T15:45:20.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:45:20.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-09-18T15:45:20.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:45:20.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-09-18T15:45:20.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:45:20.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' td/osd-scrub-repair/osd.1.log 2024-09-18T15:45:20.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:45:20.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' td/osd-scrub-repair/osd.1.log 2024-09-18T15:45:20.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1208: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-09-18T15:45:20.344 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1210: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-09-18T15:45:20.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1210: TEST_corrupt_scrub_replicated: test 1 = 1 2024-09-18T15:45:20.389 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1212: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-18T15:45:20.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1212: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-09-18T15:45:20.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1214: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-09-18T15:45:20.469 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1216: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-09-18T15:45:20.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1216: TEST_corrupt_scrub_replicated: epoch=189 2024-09-18T15:45:20.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1218: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-09-18T15:45:20.511 INFO:tasks.workunit.client.0.smithi046.stderr: . as $in 2024-09-18T15:45:20.511 INFO:tasks.workunit.client.0.smithi046.stderr: | if type == "object" then 2024-09-18T15:45:20.511 INFO:tasks.workunit.client.0.smithi046.stderr: reduce keys[] as $key 2024-09-18T15:45:20.511 INFO:tasks.workunit.client.0.smithi046.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-18T15:45:20.512 INFO:tasks.workunit.client.0.smithi046.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-18T15:45:20.512 INFO:tasks.workunit.client.0.smithi046.stderr: else f 2024-09-18T15:45:20.512 INFO:tasks.workunit.client.0.smithi046.stderr: end; 2024-09-18T15:45:20.512 INFO:tasks.workunit.client.0.smithi046.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-18T15:45:20.512 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-18T15:45:20.512 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-18T15:45:20.512 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-18T15:45:20.512 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-18T15:45:20.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1218: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-09-18T15:45:20.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1218: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-18T15:45:20.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2099: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-09-18T15:45:20.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2099: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-09-18T15:45:20.574 INFO:tasks.workunit.client.0.smithi046.stderr: . as $in 2024-09-18T15:45:20.574 INFO:tasks.workunit.client.0.smithi046.stderr: | if type == "object" then 2024-09-18T15:45:20.574 INFO:tasks.workunit.client.0.smithi046.stderr: reduce keys[] as $key 2024-09-18T15:45:20.575 INFO:tasks.workunit.client.0.smithi046.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-18T15:45:20.575 INFO:tasks.workunit.client.0.smithi046.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-18T15:45:20.575 INFO:tasks.workunit.client.0.smithi046.stderr: else f 2024-09-18T15:45:20.575 INFO:tasks.workunit.client.0.smithi046.stderr: end; 2024-09-18T15:45:20.575 INFO:tasks.workunit.client.0.smithi046.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-18T15:45:20.575 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-18T15:45:20.575 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-18T15:45:20.575 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-18T15:45:20.575 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-18T15:45:20.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2099: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-18T15:45:20.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2100: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-18T15:45:20.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-18T15:45:20.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2101: TEST_corrupt_scrub_replicated: test no = yes 2024-09-18T15:45:20.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2106: TEST_corrupt_scrub_replicated: test '' = yes 2024-09-18T15:45:20.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2111: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-09-18T15:45:20.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2113: TEST_corrupt_scrub_replicated: echo -n ZZZ 2024-09-18T15:45:20.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2114: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-09-18T15:45:20.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2116: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-18T15:45:20.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:45:20.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:45:20.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:45:20.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:45:20.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-18T15:45:20.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:45:20.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:45:20.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:45:20.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:45:20.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:45:20.727 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:45:20.727 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:45:20.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:45:20.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:45:20.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:45:21.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:45:21.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-18T15:45:21.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:45:21.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:45:21.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:45:21.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:45:21.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:45:21.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-18T15:45:22.603 INFO:tasks.workunit.client.0.smithi046.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-09-18T15:45:23.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-18T15:45:23.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:45:23.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:45:23.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:45:23.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:45:23.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:45:23.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:45:23.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:45:23.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:45:23.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:45:23.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:45:23.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:45:23.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:45:23.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:45:23.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:45:23.149 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:45:23.149 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:45:23.149 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:45:23.149 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:45:23.149 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:45:23.150 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:45:23.150 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:45:23.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:45:23.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:45:23.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:45:23.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:45:23.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:45:23.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:45:23.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:45:23.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:45:23.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:45:23.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:45:23.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:45:23.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:45:23.153 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:45:23.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:45:23.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:45:23.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:45:23.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:45:23.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:45:23.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:45:23.187 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:45:23.176+0000 7fb1b5bd0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:45:23.191 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:45:23.180+0000 7fb1b5bd0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:45:23.193 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:45:23.184+0000 7fb1b5bd0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:45:23.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:45:23.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:45:23.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:45:23.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:45:23.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:45:23.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:23.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:45:23.468 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:45:23.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:45:23.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:23.731 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:45:23.720+0000 7fb1b5bd0540 -1 Falling back to public interface 2024-09-18T15:45:23.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:24.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:24.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:24.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:45:24.854 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:45:24.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:24.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:45:25.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:26.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:26.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:26.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:45:26.181 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:45:26.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:26.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:45:26.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:26.544 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:45:26.532+0000 7fb1b5bd0540 -1 osd.0 191 log_to_monitors true 2024-09-18T15:45:27.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:27.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:27.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:45:27.490 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:45:27.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:27.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:45:27.874 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 195 up_thru 195 down_at 192 last_clean_interval [177,191) [v2:127.0.0.1:6802/393719,v1:127.0.0.1:6803/393719] [v2:127.0.0.1:6804/393719,v1:127.0.0.1:6805/393719] exists,up 537f9958-5809-4e44-8060-20ee457d6502 2024-09-18T15:45:27.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:45:27.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:45:27.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:45:27.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:45:27.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:45:27.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:45:27.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:45:27.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:45:27.876 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:45:27.877 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:45:27.877 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:45:27.878 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:45:27.878 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:45:27.878 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:45:28.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:45:28.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:45:28.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:45:28.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:45:28.109 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:45:28.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:45:28.424 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:45:28.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:45:28.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:45:28.424 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:45:28.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=837518622722 2024-09-18T15:45:28.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 837518622722 2024-09-18T15:45:28.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-837518622722' 2024-09-18T15:45:28.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:45:28.613 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:45:28.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=811748818952 2024-09-18T15:45:28.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 811748818952 2024-09-18T15:45:28.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-837518622722 1-811748818952' 2024-09-18T15:45:28.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:45:28.759 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-837518622722 2024-09-18T15:45:28.759 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:45:28.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:45:28.763 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-837518622722 2024-09-18T15:45:28.763 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:45:28.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=837518622722 2024-09-18T15:45:28.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 837518622722' 2024-09-18T15:45:28.766 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 837518622722 2024-09-18T15:45:28.767 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:45:29.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 837518622722 2024-09-18T15:45:29.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:45:30.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:45:30.087 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:45:30.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 837518622722 -lt 837518622722 2024-09-18T15:45:30.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:45:30.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-811748818952 2024-09-18T15:45:30.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:45:30.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:45:30.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-811748818952 2024-09-18T15:45:30.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:45:30.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=811748818952 2024-09-18T15:45:30.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 811748818952' 2024-09-18T15:45:30.414 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 811748818952 2024-09-18T15:45:30.415 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:45:30.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 811748818952 -lt 811748818952 2024-09-18T15:45:30.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:45:30.753 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:45:30.753 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:45:31.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:45:31.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:45:31.180 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:45:31.180 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:45:31.180 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:45:31.180 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:45:31.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:45:31.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:45:31.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:45:31.521 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:45:31.521 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:45:31.522 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:45:31.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:45:31.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:45:31.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:45:31.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2117: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-09-18T15:45:31.909 INFO:tasks.workunit.client.0.smithi046.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-09-18T15:45:31.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2119: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-09-18T15:45:31.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2120: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-09-18T15:45:31.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:45:31.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:45:31.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:45:31.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:45:31.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-09-18T15:45:31.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:45:31.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:45:31.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:45:31.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:45:31.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:45:31.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:45:31.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:45:31.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:45:31.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:45:31.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:45:32.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:45:32.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-09-18T15:45:32.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:45:32.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:45:32.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:45:32.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:45:32.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:45:32.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 get-attr _ 2024-09-18T15:45:33.746 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-18T15:45:33.746 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-18T15:45:34.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:45:34.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:45:34.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:45:34.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:45:34.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:45:34.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:45:34.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:45:34.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:45:34.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:45:34.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:45:34.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:45:34.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:45:34.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:45:34.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:45:34.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:45:34.290 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:45:34.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:45:34.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:45:34.291 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:45:34.291 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:45:34.291 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:45:34.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:45:34.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:45:34.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:45:34.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:45:34.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:45:34.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:45:34.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:45:34.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:45:34.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:45:34.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:45:34.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:45:34.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:45:34.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:45:34.294 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:45:34.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:45:34.294 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:45:34.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:45:34.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:45:34.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:45:34.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:45:34.321 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:45:34.312+0000 7fdbd0d93540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:45:34.324 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:45:34.312+0000 7fdbd0d93540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:45:34.326 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:45:34.316+0000 7fdbd0d93540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:45:34.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:45:34.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:45:34.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:45:34.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:45:34.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:45:34.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:34.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:45:34.594 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:45:34.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:34.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:45:34.867 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:45:34.856+0000 7fdbd0d93540 -1 Falling back to public interface 2024-09-18T15:45:34.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:35.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:35.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:35.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:45:35.933 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:45:35.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:35.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:45:36.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:37.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:37.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:37.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:45:37.247 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:45:37.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:37.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:45:37.557 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:45:37.548+0000 7fdbd0d93540 -1 osd.1 196 log_to_monitors true 2024-09-18T15:45:37.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:38.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:38.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:38.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:45:38.580 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:45:38.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:38.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:45:39.004 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 200 up_thru 200 down_at 197 last_clean_interval [189,196) [v2:127.0.0.1:6810/395022,v1:127.0.0.1:6811/395022] [v2:127.0.0.1:6812/395022,v1:127.0.0.1:6813/395022] exists,up 8c8956ed-6b0c-4a8b-8558-811b867c0d90 2024-09-18T15:45:39.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:45:39.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:45:39.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:45:39.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:45:39.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:45:39.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:45:39.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:45:39.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:45:39.009 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:45:39.009 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:45:39.009 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:45:39.009 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:45:39.009 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:45:39.009 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:45:39.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:45:39.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:45:39.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:45:39.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:45:39.232 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:45:39.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:45:39.541 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:45:39.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:45:39.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:45:39.542 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:45:39.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=837518622725 2024-09-18T15:45:39.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 837518622725 2024-09-18T15:45:39.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-837518622725' 2024-09-18T15:45:39.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:45:39.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:45:39.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=858993459202 2024-09-18T15:45:39.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 858993459202 2024-09-18T15:45:39.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-837518622725 1-858993459202' 2024-09-18T15:45:39.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:45:39.878 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-837518622725 2024-09-18T15:45:39.878 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:45:39.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:45:39.881 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-837518622725 2024-09-18T15:45:39.881 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:45:39.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=837518622725 2024-09-18T15:45:39.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 837518622725' 2024-09-18T15:45:39.883 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 837518622725 2024-09-18T15:45:39.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:45:40.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 837518622724 -lt 837518622725 2024-09-18T15:45:40.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:45:41.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:45:41.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:45:41.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 837518622725 -lt 837518622725 2024-09-18T15:45:41.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:45:41.532 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-858993459202 2024-09-18T15:45:41.532 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:45:41.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:45:41.535 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-858993459202 2024-09-18T15:45:41.536 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:45:41.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=858993459202 2024-09-18T15:45:41.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 858993459202' 2024-09-18T15:45:41.538 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 858993459202 2024-09-18T15:45:41.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:45:41.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 858993459202 -lt 858993459202 2024-09-18T15:45:41.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:45:41.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:45:41.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:45:42.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:45:42.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:45:42.350 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:45:42.350 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:45:42.350 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:45:42.350 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:45:42.351 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:45:42.351 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:45:42.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:45:42.690 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:45:42.690 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:45:42.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:45:43.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:45:43.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:45:43.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:45:43.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2121: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key2-ROBJ10 val2-ROBJ10 2024-09-18T15:45:43.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2122: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-18T15:45:43.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:45:43.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:45:43.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:45:43.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:45:43.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-18T15:45:43.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:45:43.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:45:43.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:45:43.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:45:43.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:45:43.168 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:45:43.168 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:45:43.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:45:43.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:45:43.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:45:44.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:45:44.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-18T15:45:44.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:45:44.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:45:44.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:45:44.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:45:44.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:45:44.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-18T15:45:46.112 INFO:tasks.workunit.client.0.smithi046.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-09-18T15:45:46.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-18T15:45:46.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:45:46.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:45:46.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:45:46.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:45:46.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:45:46.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:45:46.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:45:46.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:45:46.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:45:46.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:45:46.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:45:46.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:45:46.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:45:46.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:45:46.653 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:45:46.654 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:45:46.654 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:45:46.654 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:45:46.654 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:45:46.654 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:45:46.654 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:45:46.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:45:46.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:45:46.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:45:46.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:45:46.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:45:46.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:45:46.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:45:46.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:45:46.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:45:46.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:45:46.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:45:46.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:45:46.656 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:45:46.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:45:46.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:45:46.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:45:46.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:45:46.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:45:46.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:45:46.689 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:45:46.680+0000 7f8cad7b2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:45:46.692 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:45:46.684+0000 7f8cad7b2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:45:46.695 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:45:46.684+0000 7f8cad7b2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:45:46.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:45:46.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:45:46.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:45:46.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:45:46.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:45:46.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:46.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:45:46.969 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:45:46.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:46.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:45:47.230 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:45:47.220+0000 7f8cad7b2540 -1 Falling back to public interface 2024-09-18T15:45:47.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:48.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:48.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:48.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:45:48.331 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:45:48.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:48.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:45:48.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:49.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:49.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:49.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:45:49.640 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:45:49.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:49.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:45:49.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:50.482 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:45:50.472+0000 7f8cad7b2540 -1 osd.0 202 log_to_monitors true 2024-09-18T15:45:50.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:50.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:50.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:45:50.986 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:45:50.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:50.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:45:51.336 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 206 up_thru 200 down_at 203 last_clean_interval [195,202) [v2:127.0.0.1:6802/396345,v1:127.0.0.1:6803/396345] [v2:127.0.0.1:6804/396345,v1:127.0.0.1:6805/396345] exists,up 537f9958-5809-4e44-8060-20ee457d6502 2024-09-18T15:45:51.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:45:51.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:45:51.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:45:51.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:45:51.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:45:51.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:45:51.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:45:51.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:45:51.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:45:51.338 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:45:51.338 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:45:51.339 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:45:51.339 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:45:51.339 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:45:51.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:45:51.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:45:51.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:45:51.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:45:51.564 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:45:51.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:45:51.957 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:45:51.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:45:51.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:45:51.958 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:45:52.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=884763262978 2024-09-18T15:45:52.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 884763262978 2024-09-18T15:45:52.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-884763262978' 2024-09-18T15:45:52.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:45:52.132 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:45:52.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=858993459205 2024-09-18T15:45:52.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 858993459205 2024-09-18T15:45:52.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-884763262978 1-858993459205' 2024-09-18T15:45:52.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:45:52.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-884763262978 2024-09-18T15:45:52.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:45:52.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:45:52.282 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-884763262978 2024-09-18T15:45:52.282 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:45:52.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=884763262978 2024-09-18T15:45:52.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 884763262978' 2024-09-18T15:45:52.284 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 884763262978 2024-09-18T15:45:52.285 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:45:52.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 884763262978 2024-09-18T15:45:52.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:45:53.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:45:53.593 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:45:53.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 884763262978 -lt 884763262978 2024-09-18T15:45:53.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:45:53.952 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-858993459205 2024-09-18T15:45:53.952 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:45:53.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:45:53.956 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-858993459205 2024-09-18T15:45:53.956 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:45:53.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=858993459205 2024-09-18T15:45:53.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 858993459205' 2024-09-18T15:45:53.959 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 858993459205 2024-09-18T15:45:53.959 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:45:54.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 858993459206 -lt 858993459205 2024-09-18T15:45:54.258 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:45:54.259 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:45:54.259 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:45:54.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:45:54.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:45:54.676 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:45:54.676 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:45:54.676 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:45:54.676 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:45:54.677 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:45:54.677 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:45:55.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:45:55.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:45:55.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:45:55.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:45:55.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:45:55.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:45:55.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:45:55.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2123: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-18T15:45:55.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:45:55.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:45:55.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:45:55.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:45:55.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-18T15:45:55.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:45:55.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:45:55.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:45:55.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:45:55.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:45:55.394 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:45:55.394 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:45:55.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:45:55.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:45:55.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:45:55.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:45:55.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-18T15:45:55.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:45:55.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:45:55.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:45:55.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:45:55.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:45:55.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-18T15:45:56.979 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-18T15:45:56.979 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-18T15:45:57.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:45:57.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:45:57.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:45:57.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:45:57.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:45:57.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:45:57.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:45:57.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:45:57.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:45:57.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:45:57.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:45:57.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:45:57.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:45:57.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:45:57.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:45:57.517 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:45:57.517 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:45:57.517 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:45:57.519 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:45:57.519 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:45:57.519 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:45:57.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:45:57.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:45:57.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:45:57.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:45:57.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:45:57.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:45:57.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:45:57.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:45:57.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:45:57.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:45:57.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:45:57.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:45:57.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:45:57.520 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:45:57.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:45:57.521 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:45:57.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:45:57.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:45:57.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:45:57.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:45:57.554 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:45:57.544+0000 7f60c6d46540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:45:57.557 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:45:57.548+0000 7f60c6d46540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:45:57.560 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:45:57.548+0000 7f60c6d46540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:45:57.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:45:57.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:45:57.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:45:57.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:45:57.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:45:57.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:57.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:45:57.843 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:45:57.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:57.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:45:58.095 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:45:58.084+0000 7f60c6d46540 -1 Falling back to public interface 2024-09-18T15:45:58.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:59.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:59.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:59.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:45:59.142 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:45:59.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:59.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:45:59.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:00.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:00.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:00.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:46:00.457 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:46:00.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:46:00.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:00.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:00.988 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:46:00.976+0000 7f60c6d46540 -1 osd.1 208 log_to_monitors true 2024-09-18T15:46:01.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:01.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:01.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:46:01.766 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:46:01.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:01.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:46:02.316 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 212 up_thru 212 down_at 209 last_clean_interval [200,208) [v2:127.0.0.1:6810/397647,v1:127.0.0.1:6811/397647] [v2:127.0.0.1:6812/397647,v1:127.0.0.1:6813/397647] exists,up 8c8956ed-6b0c-4a8b-8558-811b867c0d90 2024-09-18T15:46:02.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:46:02.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:46:02.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:46:02.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:46:02.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:46:02.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:46:02.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:46:02.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:46:02.317 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:46:02.318 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:46:02.318 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:46:02.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:46:02.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:46:02.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:46:02.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:46:02.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:46:02.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:46:02.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:46:02.555 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:46:02.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:46:02.878 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:46:02.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:46:02.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:46:02.878 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:46:03.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=884763262981 2024-09-18T15:46:03.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 884763262981 2024-09-18T15:46:03.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-884763262981' 2024-09-18T15:46:03.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:46:03.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:46:03.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=910533066754 2024-09-18T15:46:03.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 910533066754 2024-09-18T15:46:03.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-884763262981 1-910533066754' 2024-09-18T15:46:03.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:46:03.214 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-884763262981 2024-09-18T15:46:03.214 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:46:03.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:46:03.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-884763262981 2024-09-18T15:46:03.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:46:03.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=884763262981 2024-09-18T15:46:03.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 884763262981' 2024-09-18T15:46:03.219 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 884763262981 2024-09-18T15:46:03.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:46:03.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 884763262981 -lt 884763262981 2024-09-18T15:46:03.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:46:03.529 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-910533066754 2024-09-18T15:46:03.529 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:46:03.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:46:03.533 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-910533066754 2024-09-18T15:46:03.533 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:46:03.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=910533066754 2024-09-18T15:46:03.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 910533066754' 2024-09-18T15:46:03.536 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 910533066754 2024-09-18T15:46:03.536 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:46:03.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 910533066754 2024-09-18T15:46:03.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:46:04.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:46:04.938 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:46:05.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 910533066754 -lt 910533066754 2024-09-18T15:46:05.257 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:46:05.257 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:46:05.258 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:46:05.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:46:05.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:46:05.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:46:05.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:46:05.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:46:05.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:46:05.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:46:05.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:46:05.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:46:05.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:46:05.973 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:46:05.973 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:46:06.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:46:06.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:46:06.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:46:06.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2124: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi 2024-09-18T15:46:06.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2126: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-09-18T15:46:06.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:46:06.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:46:06.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:46:06.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:46:06.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=csr_pool 2024-09-18T15:46:06.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:46:06.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=ROBJ11 2024-09-18T15:46:06.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:46:06.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-scrub-repair 2024-09-18T15:46:06.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:46:06.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:46:06.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:46:06.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T15:46:06.347 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds csr_pool ROBJ11 2024-09-18T15:46:06.347 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-09-18T15:46:06.347 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ11 2024-09-18T15:46:06.347 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-09-18T15:46:06.348 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:46:06.696 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-09-18T15:46:06.696 INFO:tasks.workunit.client.0.smithi046.stderr:0' 2024-09-18T15:46:06.696 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-09-18T15:46:06.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:46:06.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-18T15:46:06.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:46:06.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:46:06.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-scrub-repair/1/type 2024-09-18T15:46:06.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:46:06.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-18T15:46:06.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:46:06.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T15:46:06.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:46:06.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:46:06.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:46:06.701 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T15:46:06.701 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:46:06.701 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:46:06.702 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:46:06.702 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:46:06.702 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:46:06.703 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-osd.1.asok 2024-09-18T15:46:06.703 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.137209/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:46:06.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:46:06.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:46:06.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:46:06.802 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-18T15:46:06.802 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:46:06.802 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:46:06.803 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:46:06.803 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:46:06.803 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:46:06.803 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-osd.1.asok 2024-09-18T15:46:06.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:46:06.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.137209/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-09-18T15:46:06.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2127: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-09-18T15:46:06.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:46:06.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:46:06.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=mdata 2024-09-18T15:46:06.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:46:06.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=csr_pool 2024-09-18T15:46:06.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:46:06.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=ROBJ12 2024-09-18T15:46:06.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:46:06.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-scrub-repair 2024-09-18T15:46:06.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:46:06.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-18T15:46:06.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:46:06.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T15:46:06.896 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds csr_pool ROBJ12 2024-09-18T15:46:06.896 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-09-18T15:46:06.896 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ12 2024-09-18T15:46:06.896 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-09-18T15:46:06.896 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:46:07.199 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-09-18T15:46:07.200 INFO:tasks.workunit.client.0.smithi046.stderr:0' 2024-09-18T15:46:07.200 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-09-18T15:46:07.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:46:07.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=0 2024-09-18T15:46:07.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:46:07.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:46:07.201 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-scrub-repair/0/type 2024-09-18T15:46:07.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:46:07.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-18T15:46:07.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:46:07.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-09-18T15:46:07.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:46:07.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:46:07.204 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:46:07.205 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-09-18T15:46:07.205 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T15:46:07.205 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T15:46:07.206 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:46:07.206 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:46:07.206 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:46:07.206 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-osd.0.asok 2024-09-18T15:46:07.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.137209/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:46:07.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:46:07.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:46:07.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:46:07.305 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.0 2024-09-18T15:46:07.305 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T15:46:07.305 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T15:46:07.305 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:46:07.305 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:46:07.306 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:46:07.306 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-osd.0.asok 2024-09-18T15:46:07.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:46:07.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.137209/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-09-18T15:46:07.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2128: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-09-18T15:46:07.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:46:07.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:46:07.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=mdata 2024-09-18T15:46:07.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:46:07.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=csr_pool 2024-09-18T15:46:07.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:46:07.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=ROBJ13 2024-09-18T15:46:07.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:46:07.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-scrub-repair 2024-09-18T15:46:07.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:46:07.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-18T15:46:07.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:46:07.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T15:46:07.406 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds csr_pool ROBJ13 2024-09-18T15:46:07.406 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-09-18T15:46:07.406 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-09-18T15:46:07.407 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-09-18T15:46:07.407 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:46:07.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-09-18T15:46:07.709 INFO:tasks.workunit.client.0.smithi046.stderr:0' 2024-09-18T15:46:07.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-09-18T15:46:07.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:46:07.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=0 2024-09-18T15:46:07.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:46:07.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:46:07.710 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-scrub-repair/0/type 2024-09-18T15:46:07.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:46:07.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-18T15:46:07.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:46:07.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-09-18T15:46:07.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:46:07.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:46:07.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:46:07.714 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-09-18T15:46:07.714 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T15:46:07.714 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T15:46:07.715 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:46:07.715 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:46:07.715 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:46:07.716 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-osd.0.asok 2024-09-18T15:46:07.716 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.137209/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:46:07.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:46:07.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:46:07.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:46:07.811 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.0 2024-09-18T15:46:07.811 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T15:46:07.811 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T15:46:07.812 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:46:07.812 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:46:07.812 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:46:07.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-osd.0.asok 2024-09-18T15:46:07.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:46:07.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.137209/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-09-18T15:46:07.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2129: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-09-18T15:46:07.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:46:07.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:46:07.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:46:07.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:46:07.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=csr_pool 2024-09-18T15:46:07.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:46:07.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=ROBJ13 2024-09-18T15:46:07.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:46:07.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-scrub-repair 2024-09-18T15:46:07.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:46:07.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:46:07.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:46:07.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T15:46:07.906 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds csr_pool ROBJ13 2024-09-18T15:46:07.907 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-09-18T15:46:07.907 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-09-18T15:46:07.907 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-09-18T15:46:07.907 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:46:08.212 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-09-18T15:46:08.212 INFO:tasks.workunit.client.0.smithi046.stderr:0' 2024-09-18T15:46:08.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-09-18T15:46:08.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:46:08.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-18T15:46:08.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:46:08.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:46:08.214 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-scrub-repair/1/type 2024-09-18T15:46:08.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:46:08.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-18T15:46:08.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:46:08.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T15:46:08.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:46:08.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:46:08.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:46:08.217 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T15:46:08.217 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:46:08.217 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:46:08.218 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:46:08.218 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:46:08.218 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:46:08.218 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-osd.1.asok 2024-09-18T15:46:08.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.137209/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:46:08.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:46:08.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:46:08.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:46:08.315 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-18T15:46:08.315 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:46:08.315 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:46:08.316 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:46:08.316 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:46:08.316 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:46:08.316 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-osd.1.asok 2024-09-18T15:46:08.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:46:08.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.137209/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-09-18T15:46:08.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2132: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=134217728 2024-09-18T15:46:08.537 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0: {} 2024-09-18T15:46:08.557 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1: {} 2024-09-18T15:46:08.559 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0: osd_max_object_size = '' 2024-09-18T15:46:08.559 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1: osd_max_object_size = '' 2024-09-18T15:46:08.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2134: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-09-18T15:46:08.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: pg_deep_scrub: local pgid=3.0 2024-09-18T15:46:08.574 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-18T15:46:08.574 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:46:08.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:46:08.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:46:08.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-18T15:46:08.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: pg_deep_scrub: local last_scrub=2024-09-18T15:40:57.142428+0000 2024-09-18T15:46:08.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-09-18T15:46:09.174 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-09-18T15:46:09.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: pg_deep_scrub: wait_for_scrub 3.0 2024-09-18T15:40:57.142428+0000 last_deep_scrub_stamp 2024-09-18T15:46:09.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=3.0 2024-09-18T15:46:09.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:40:57.142428+0000 2024-09-18T15:46:09.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-09-18T15:46:09.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:46:09.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:46:09.189 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-18T15:46:09.189 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:46:09.190 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:46:09.190 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:46:09.190 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-18T15:46:09.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:40:57.142428+0000 '>' 2024-09-18T15:40:57.142428+0000 2024-09-18T15:46:09.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:46:10.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:46:10.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:46:10.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-18T15:46:10.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:46:10.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:46:10.501 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:46:10.501 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-18T15:46:10.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:40:57.142428+0000 '>' 2024-09-18T15:40:57.142428+0000 2024-09-18T15:46:10.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:46:11.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:46:11.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:46:11.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-18T15:46:11.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:46:11.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:46:11.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:46:11.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-18T15:46:12.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:40:57.142428+0000 '>' 2024-09-18T15:40:57.142428+0000 2024-09-18T15:46:12.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:46:13.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:46:13.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:46:13.135 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-18T15:46:13.135 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:46:13.136 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:46:13.136 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:46:13.137 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-18T15:46:14.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:46:10.023291+0000 '>' 2024-09-18T15:40:57.142428+0000 2024-09-18T15:46:14.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:46:14.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2136: TEST_corrupt_scrub_replicated: err_strings=() 2024-09-18T15:46:14.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2137: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-09-18T15:46:14.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2138: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' 2024-09-18T15:46:14.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2139: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' 2024-09-18T15:46:14.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2140: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' 2024-09-18T15:46:14.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2141: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-09-18T15:46:14.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2142: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-09-18T15:46:14.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2143: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-09-18T15:46:14.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2144: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-09-18T15:46:14.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2145: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' 2024-09-18T15:46:14.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2146: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-09-18T15:46:14.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2147: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-09-18T15:46:14.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2148: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' 2024-09-18T15:46:14.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2149: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' 2024-09-18T15:46:14.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2150: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' 2024-09-18T15:46:14.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2151: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' 2024-09-18T15:46:14.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2152: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-09-18T15:46:14.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2153: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-09-18T15:46:14.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2154: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' 2024-09-18T15:46:14.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2155: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-09-18T15:46:14.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2156: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-09-18T15:46:14.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2157: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-09-18T15:46:14.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2158: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-09-18T15:46:14.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2159: TEST_corrupt_scrub_replicated: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-09-18T15:46:14.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: err_strings[23]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-09-18T15:46:14.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2161: TEST_corrupt_scrub_replicated: err_strings[24]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' 2024-09-18T15:46:14.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: err_strings[25]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-09-18T15:46:14.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2163: TEST_corrupt_scrub_replicated: err_strings[26]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' 2024-09-18T15:46:14.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2164: TEST_corrupt_scrub_replicated: err_strings[27]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' 2024-09-18T15:46:14.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2165: TEST_corrupt_scrub_replicated: err_strings[28]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-09-18T15:46:14.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2166: TEST_corrupt_scrub_replicated: err_strings[29]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' 2024-09-18T15:46:14.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2167: TEST_corrupt_scrub_replicated: err_strings[30]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' 2024-09-18T15:46:14.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2168: TEST_corrupt_scrub_replicated: err_strings[31]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' 2024-09-18T15:46:14.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2169: TEST_corrupt_scrub_replicated: err_strings[32]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' 2024-09-18T15:46:14.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2170: TEST_corrupt_scrub_replicated: err_strings[33]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-09-18T15:46:14.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: err_strings[34]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-09-18T15:46:14.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2172: TEST_corrupt_scrub_replicated: err_strings[35]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' 2024-09-18T15:46:14.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: err_strings[36]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-09-18T15:46:14.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2174: TEST_corrupt_scrub_replicated: err_strings[37]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' 2024-09-18T15:46:14.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: err_strings[38]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' 2024-09-18T15:46:14.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-18T15:46:14.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' td/osd-scrub-repair/osd.1.log 2024-09-18T15:46:14.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2186: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-09-18T15:46:14.715 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2188: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-09-18T15:46:14.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2188: TEST_corrupt_scrub_replicated: test 1 = 1 2024-09-18T15:46:14.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2190: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-18T15:46:14.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2190: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-09-18T15:46:14.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2192: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-09-18T15:46:14.844 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2194: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-09-18T15:46:14.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2194: TEST_corrupt_scrub_replicated: epoch=212 2024-09-18T15:46:14.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2196: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-09-18T15:46:14.886 INFO:tasks.workunit.client.0.smithi046.stderr: . as $in 2024-09-18T15:46:14.886 INFO:tasks.workunit.client.0.smithi046.stderr: | if type == "object" then 2024-09-18T15:46:14.886 INFO:tasks.workunit.client.0.smithi046.stderr: reduce keys[] as $key 2024-09-18T15:46:14.886 INFO:tasks.workunit.client.0.smithi046.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-18T15:46:14.886 INFO:tasks.workunit.client.0.smithi046.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-18T15:46:14.886 INFO:tasks.workunit.client.0.smithi046.stderr: else f 2024-09-18T15:46:14.886 INFO:tasks.workunit.client.0.smithi046.stderr: end; 2024-09-18T15:46:14.886 INFO:tasks.workunit.client.0.smithi046.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-18T15:46:14.887 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-18T15:46:14.887 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-18T15:46:14.887 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-18T15:46:14.887 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-18T15:46:14.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2196: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-09-18T15:46:14.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2196: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-18T15:46:14.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3575: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-09-18T15:46:14.959 INFO:tasks.workunit.client.0.smithi046.stderr: . as $in 2024-09-18T15:46:14.959 INFO:tasks.workunit.client.0.smithi046.stderr: | if type == "object" then 2024-09-18T15:46:14.959 INFO:tasks.workunit.client.0.smithi046.stderr: reduce keys[] as $key 2024-09-18T15:46:14.960 INFO:tasks.workunit.client.0.smithi046.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-18T15:46:14.960 INFO:tasks.workunit.client.0.smithi046.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-18T15:46:14.960 INFO:tasks.workunit.client.0.smithi046.stderr: else f 2024-09-18T15:46:14.960 INFO:tasks.workunit.client.0.smithi046.stderr: end; 2024-09-18T15:46:14.960 INFO:tasks.workunit.client.0.smithi046.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-18T15:46:14.960 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-18T15:46:14.960 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-18T15:46:14.960 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-18T15:46:14.960 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-18T15:46:14.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3575: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-09-18T15:46:14.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3575: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-18T15:46:15.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3576: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-18T15:46:15.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-18T15:46:15.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3577: TEST_corrupt_scrub_replicated: test no = yes 2024-09-18T15:46:15.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3582: TEST_corrupt_scrub_replicated: test '' = yes 2024-09-18T15:46:15.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3587: TEST_corrupt_scrub_replicated: repair 3.0 2024-09-18T15:46:15.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=3.0 2024-09-18T15:46:15.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 3.0 2024-09-18T15:46:15.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:46:15.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:46:15.027 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:46:15.027 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:46:15.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T15:46:10.023291+0000 2024-09-18T15:46:15.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 3.0 2024-09-18T15:46:15.621 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 3.0 on osd.1 to repair 2024-09-18T15:46:15.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 3.0 2024-09-18T15:46:10.023291+0000 2024-09-18T15:46:15.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=3.0 2024-09-18T15:46:15.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:46:10.023291+0000 2024-09-18T15:46:15.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:46:15.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:46:15.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:46:15.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:46:15.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:46:15.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:46:15.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:46:15.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:46:15.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:46:10.023291+0000 '>' 2024-09-18T15:46:10.023291+0000 2024-09-18T15:46:15.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:46:16.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:46:16.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:46:16.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:46:16.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:46:16.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:46:16.936 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:46:16.937 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:46:17.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:46:10.023291+0000 '>' 2024-09-18T15:46:10.023291+0000 2024-09-18T15:46:17.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:46:18.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:46:18.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:46:18.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:46:18.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:46:18.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:46:18.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:46:18.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:46:18.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:46:10.023291+0000 '>' 2024-09-18T15:46:10.023291+0000 2024-09-18T15:46:18.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:46:19.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:46:19.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:46:19.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:46:19.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:46:19.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:46:19.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:46:19.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:46:19.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:46:10.023291+0000 '>' 2024-09-18T15:46:10.023291+0000 2024-09-18T15:46:19.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:46:20.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:46:20.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:46:20.896 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:46:20.896 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:46:20.896 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:46:20.896 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:46:20.897 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:46:21.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:46:10.023291+0000 '>' 2024-09-18T15:46:10.023291+0000 2024-09-18T15:46:21.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:46:22.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:46:22.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:46:22.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:46:22.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:46:22.214 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:46:22.214 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:46:22.214 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:46:22.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:46:10.023291+0000 '>' 2024-09-18T15:46:10.023291+0000 2024-09-18T15:46:22.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:46:23.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:46:23.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:46:23.529 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:46:23.529 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:46:23.530 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:46:23.530 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:46:23.530 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:46:23.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:46:21.028987+0000 '>' 2024-09-18T15:46:10.023291+0000 2024-09-18T15:46:23.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:46:23.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3588: TEST_corrupt_scrub_replicated: wait_for_clean 2024-09-18T15:46:23.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:46:23.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:46:23.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:46:23.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:46:23.851 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:46:23.851 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:46:23.851 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:46:23.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:46:23.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:46:23.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:46:24.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:46:24.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:46:24.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:46:24.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:46:24.083 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:46:24.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:46:24.391 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:46:24.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:46:24.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:46:24.392 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:46:24.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=884763262986 2024-09-18T15:46:24.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 884763262986 2024-09-18T15:46:24.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-884763262986' 2024-09-18T15:46:24.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:46:24.565 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:46:24.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=910533066759 2024-09-18T15:46:24.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 910533066759 2024-09-18T15:46:24.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-884763262986 1-910533066759' 2024-09-18T15:46:24.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:46:24.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-884763262986 2024-09-18T15:46:24.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:46:24.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:46:24.716 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-884763262986 2024-09-18T15:46:24.716 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:46:24.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=884763262986 2024-09-18T15:46:24.719 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 884763262986 2024-09-18T15:46:24.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 884763262986' 2024-09-18T15:46:24.719 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:46:25.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 884763262985 -lt 884763262986 2024-09-18T15:46:25.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:46:26.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:46:26.046 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:46:26.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 884763262986 -lt 884763262986 2024-09-18T15:46:26.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:46:26.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-910533066759 2024-09-18T15:46:26.369 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:46:26.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:46:26.372 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-910533066759 2024-09-18T15:46:26.372 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:46:26.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=910533066759 2024-09-18T15:46:26.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 910533066759' 2024-09-18T15:46:26.375 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 910533066759 2024-09-18T15:46:26.376 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:46:26.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 910533066759 -lt 910533066759 2024-09-18T15:46:26.718 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:46:26.719 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:46:26.719 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:46:27.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:46:27.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:46:27.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:46:27.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:46:27.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:46:27.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:46:27.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:46:27.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:46:27.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:46:27.430 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:46:27.431 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:46:27.431 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:46:27.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:46:27.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:46:27.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:46:27.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3591: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ17 td/osd-scrub-repair/robj17.out 2024-09-18T15:46:27.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3592: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ18 td/osd-scrub-repair/robj18.out 2024-09-18T15:46:27.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3594: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-09-18T15:46:27.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3595: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-09-18T15:46:27.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-09-18T15:46:27.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-09-18T15:46:27.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: TEST_corrupt_scrub_replicated: '[' 0 '!=' 0 ']' 2024-09-18T15:46:27.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3605: TEST_corrupt_scrub_replicated: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-09-18T15:46:28.369 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'csr_pool' removed 2024-09-18T15:46:28.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T15:46:28.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:46:28.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:46:28.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:46:28.384 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:46:28.384 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:46:28.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:46:28.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:46:28.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:46:28.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:46:28.504 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:46:28.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:46:28.507 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:46:28.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:46:28.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:46:28.511 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:46:28.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:46:28.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:46:28.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:46:28.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:46:28.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:46:28.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:46:28.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:46:28.561 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:46:28.561 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:46:28.561 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:46:28.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:46:28.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:46:28.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:46:28.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T15:46:28.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T15:46:28.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T15:46:28.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T15:46:28.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:46:28.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:46:28.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:46:28.565 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:46:28.565 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:46:28.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:46:28.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:46:28.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:46:28.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:46:28.572 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:46:28.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:46:28.574 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:46:28.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:46:28.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:46:28.577 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:46:28.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:46:28.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:46:28.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:46:28.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:46:28.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:46:28.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:46:28.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:46:28.588 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:46:28.588 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:46:28.588 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:46:28.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:46:28.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:46:28.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:46:28.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T15:46:28.593 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:46:28.593 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:46:28.593 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:46:28.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T15:46:28.596 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:46:28.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:46:28.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:46:28.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T15:46:28.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_corrupt_snapset_scrub_rep td/osd-scrub-repair 2024-09-18T15:46:28.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5952: TEST_corrupt_snapset_scrub_rep: local dir=td/osd-scrub-repair 2024-09-18T15:46:28.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5953: TEST_corrupt_snapset_scrub_rep: local poolname=csr_pool 2024-09-18T15:46:28.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: local total_objs=2 2024-09-18T15:46:28.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5956: TEST_corrupt_snapset_scrub_rep: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-09-18T15:46:28.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T15:46:28.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:46:28.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:46:28.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:46:28.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T15:46:28.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-09-18T15:46:28.988 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:46:28.988 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:46:28.988 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:46:28.989 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:46:28.989 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:46:28.989 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:46:28.990 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:46:28.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-18T15:46:29.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:46:29.287 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:46:29.287 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:46:29.287 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:46:29.287 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:46:29.288 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:46:29.289 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:46:29.289 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:46:29.289 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:46:29.290 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:46:29.290 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:46:29.290 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:46:29.291 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:46:29.292 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:46:29.292 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T15:46:29.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:46:29.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:46:29.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:46:29.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:46:29.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:46:29.386 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:46:29.386 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:46:29.386 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:46:29.387 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:46:29.387 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:46:29.387 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:46:29.387 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:46:29.388 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:46:29.388 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T15:46:29.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5957: TEST_corrupt_snapset_scrub_rep: run_mgr td/osd-scrub-repair x 2024-09-18T15:46:29.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T15:46:29.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:46:29.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:46:29.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:46:29.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T15:46:29.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:46:30.339 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:46:30.339 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:46:30.339 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:46:30.340 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:46:30.340 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:46:30.340 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:46:30.341 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:46:30.342 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:46:30.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:46:30.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5958: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 0 2024-09-18T15:46:30.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:46:30.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:46:30.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:46:30.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:46:30.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:46:30.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:46:30.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:46:30.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:46:30.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:46:30.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:46:30.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:46:30.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:46:30.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:46:30.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:46:30.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:46:30.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:46:30.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:46:30.383 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:46:30.383 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:46:30.383 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:46:30.384 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:46:30.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:46:30.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:46:30.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:46:30.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:46:30.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:46:30.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:46:30.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:46:30.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:46:30.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:46:30.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:46:30.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:46:30.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:46:30.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:46:30.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:46:30.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f5b28f60-d48d-403c-a22a-a35a92723ee3 2024-09-18T15:46:30.389 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 f5b28f60-d48d-403c-a22a-a35a92723ee3 2024-09-18T15:46:30.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f5b28f60-d48d-403c-a22a-a35a92723ee3' 2024-09-18T15:46:30.389 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:46:30.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDW9epm0pT4FxAAjayQSvBDE1r7SB2R+9qyNw== 2024-09-18T15:46:30.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDW9epm0pT4FxAAjayQSvBDE1r7SB2R+9qyNw=="}' 2024-09-18T15:46:30.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f5b28f60-d48d-403c-a22a-a35a92723ee3 -i td/osd-scrub-repair/0/new.json 2024-09-18T15:46:30.686 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:46:30.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-18T15:46:30.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDW9epm0pT4FxAAjayQSvBDE1r7SB2R+9qyNw== --osd-uuid f5b28f60-d48d-403c-a22a-a35a92723ee3 2024-09-18T15:46:30.730 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:46:30.720+0000 7f6ed169d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:46:30.733 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:46:30.724+0000 7f6ed169d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:46:30.735 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:46:30.724+0000 7f6ed169d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:46:30.736 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:46:30.724+0000 7f6ed169d540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:46:30.736 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:46:30.728+0000 7f6ed169d540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:46:30.736 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:46:30.728+0000 7f6ed169d540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:46:30.819 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:46:30.808+0000 7f6ed169d540 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-18T15:46:36.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T15:46:36.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:46:36.795 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T15:46:36.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:46:36.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:46:37.174 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T15:46:37.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:46:37.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:46:37.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:46:37.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:46:37.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:46:37.206 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:46:37.196+0000 7f76cc0e6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:46:37.208 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:46:37.200+0000 7f76cc0e6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:46:37.210 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:46:37.200+0000 7f76cc0e6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:46:37.495 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:46:37.484+0000 7f76cc0e6540 -1 Falling back to public interface 2024-09-18T15:46:37.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:46:37.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:46:37.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:46:37.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:46:37.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:46:37.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:37.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:46:37.522 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:46:37.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:37.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:46:37.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:38.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:38.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:38.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:46:38.821 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:46:38.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:38.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:46:39.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:40.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:40.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:40.142 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:46:40.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:46:40.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:40.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:46:40.254 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:46:40.244+0000 7f76cc0e6540 -1 osd.0 0 log_to_monitors true 2024-09-18T15:46:40.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:41.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:41.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:41.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:46:41.503 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:46:41.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:41.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:46:41.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:42.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:42.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:42.824 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:46:42.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:46:42.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:42.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:46:43.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:44.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:44.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:44.138 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:46:44.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:46:44.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:44.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:46:44.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:45.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:45.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:45.455 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:46:45.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:46:45.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:45.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:46:45.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:46.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:46.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:46.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:46:46.773 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:46:46.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:46.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:46:47.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:48.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:48.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:48.122 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:46:48.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:46:48.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:48.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:46:48.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:49.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:49.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:49.440 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:46:49.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:46:49.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:49.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:46:49.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:50.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:50.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:50.756 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:46:50.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:46:50.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:50.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:46:51.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:52.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:52.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:52.134 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:46:52.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:46:52.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:52.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:46:52.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:53.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:53.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:53.470 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:46:53.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:46:53.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:53.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:46:54.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:55.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:55.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:55.019 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:46:55.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:46:55.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:55.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:46:55.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:56.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:56.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:56.668 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:46:56.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:46:56.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:56.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:46:57.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:58.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:58.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:58.290 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:46:58.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:46:58.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:58.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:46:58.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:59.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:59.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:59.800 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:46:59.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:46:59.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:59.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:47:00.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:01.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:01.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:01.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:47:01.568 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:47:01.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:01.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:47:01.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:02.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:02.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:02.882 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:47:02.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:47:02.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:02.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:47:03.204 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/400732,v1:127.0.0.1:6803/400732] [v2:127.0.0.1:6804/400732,v1:127.0.0.1:6805/400732] exists,up f5b28f60-d48d-403c-a22a-a35a92723ee3 2024-09-18T15:47:03.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:47:03.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:47:03.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:47:03.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5959: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 1 2024-09-18T15:47:03.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:47:03.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:47:03.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:47:03.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:47:03.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:47:03.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:47:03.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:47:03.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:47:03.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:47:03.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:47:03.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:47:03.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:47:03.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:47:03.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:47:03.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:47:03.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:47:03.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:47:03.210 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:47:03.210 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:47:03.210 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:47:03.210 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:47:03.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:47:03.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:47:03.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:47:03.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:47:03.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:47:03.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:47:03.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:47:03.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:47:03.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:47:03.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:47:03.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:47:03.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:47:03.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:47:03.212 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:47:03.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7a06f774-5cc7-45a2-8c6c-0e7441aa66fe 2024-09-18T15:47:03.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7a06f774-5cc7-45a2-8c6c-0e7441aa66fe' 2024-09-18T15:47:03.214 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 7a06f774-5cc7-45a2-8c6c-0e7441aa66fe 2024-09-18T15:47:03.215 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:47:03.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD39epmKZrODRAAXyyGm/+LsbovUgWqViRqDw== 2024-09-18T15:47:03.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD39epmKZrODRAAXyyGm/+LsbovUgWqViRqDw=="}' 2024-09-18T15:47:03.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7a06f774-5cc7-45a2-8c6c-0e7441aa66fe -i td/osd-scrub-repair/1/new.json 2024-09-18T15:47:03.685 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:47:03.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-18T15:47:03.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD39epmKZrODRAAXyyGm/+LsbovUgWqViRqDw== --osd-uuid 7a06f774-5cc7-45a2-8c6c-0e7441aa66fe 2024-09-18T15:47:03.732 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:47:03.724+0000 7fd7698d5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:47:03.742 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:47:03.736+0000 7fd7698d5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:47:03.743 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:47:03.736+0000 7fd7698d5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:47:03.745 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:47:03.740+0000 7fd7698d5540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:47:03.746 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:47:03.740+0000 7fd7698d5540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:47:03.746 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:47:03.740+0000 7fd7698d5540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:47:03.826 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:47:03.820+0000 7fd7698d5540 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-18T15:47:09.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-18T15:47:09.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:47:09.898 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T15:47:09.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:47:09.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:47:10.286 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T15:47:10.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:47:10.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:47:10.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:47:10.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:47:10.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:47:10.318 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:47:10.312+0000 7f4768bbe540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:47:10.321 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:47:10.316+0000 7f4768bbe540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:47:10.322 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:47:10.316+0000 7f4768bbe540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:47:10.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:47:10.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:47:10.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:47:10.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:47:10.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:47:10.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:10.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:47:10.635 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:47:10.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:10.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:47:10.858 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:47:10.852+0000 7f4768bbe540 -1 Falling back to public interface 2024-09-18T15:47:10.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:11.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:11.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:11.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:47:11.968 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:47:11.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:11.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:47:12.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:13.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:13.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:13.283 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:47:13.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:47:13.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:13.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:47:13.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:13.604 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:47:13.596+0000 7f4768bbe540 -1 osd.1 0 log_to_monitors true 2024-09-18T15:47:14.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:14.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:14.593 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:47:14.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:47:14.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:14.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:47:14.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:15.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:15.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:15.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:47:15.917 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:47:15.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:15.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:47:16.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:17.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:17.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:17.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:47:17.232 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:47:17.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:17.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:47:17.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:18.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:18.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:18.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:47:18.591 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:47:18.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:18.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:47:19.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:20.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:20.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:20.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:47:20.030 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:47:20.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:20.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:47:20.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:21.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:21.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:21.413 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:47:21.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:47:21.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:21.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:47:21.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:22.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:22.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:22.728 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:47:22.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:47:22.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:22.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:47:23.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:24.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:24.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:24.044 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:47:24.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:47:24.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:24.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:47:24.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:25.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:25.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:25.363 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:47:25.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:47:25.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:25.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:47:25.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:26.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:26.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:26.677 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:47:26.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:47:26.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:26.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:47:27.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:28.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:28.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:28.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:47:28.314 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:47:28.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:28.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:47:28.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:29.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:29.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:29.662 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:47:29.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:47:29.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:29.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:47:30.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:31.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:31.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:31.444 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:47:31.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:47:31.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:31.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:47:32.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:33.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:33.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:33.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:47:33.262 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:47:33.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:33.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:47:33.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:34.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:34.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:34.770 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:47:34.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:47:34.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:34.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:47:35.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:36.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:36.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:36.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:47:36.550 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:47:36.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:36.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:47:37.005 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/402563,v1:127.0.0.1:6811/402563] [v2:127.0.0.1:6812/402563,v1:127.0.0.1:6813/402563] exists,up 7a06f774-5cc7-45a2-8c6c-0e7441aa66fe 2024-09-18T15:47:37.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:47:37.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:47:37.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:47:37.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: create_rbd_pool 2024-09-18T15:47:37.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T15:47:37.341 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' does not exist 2024-09-18T15:47:37.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T15:47:37.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T15:47:37.838 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' created 2024-09-18T15:47:37.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:47:38.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T15:47:40.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-09-18T15:47:40.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:47:40.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:47:40.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:47:40.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:47:40.858 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:47:40.859 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:47:40.860 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:47:40.860 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:47:40.860 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:47:40.860 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:47:41.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:47:41.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:47:41.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:47:41.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:47:41.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:47:41.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:47:41.422 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:47:41.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:47:41.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:47:41.423 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:47:41.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836492 2024-09-18T15:47:41.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836492 2024-09-18T15:47:41.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836492' 2024-09-18T15:47:41.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:47:41.608 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:47:41.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672965 2024-09-18T15:47:41.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672965 2024-09-18T15:47:41.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836492 1-42949672965' 2024-09-18T15:47:41.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:47:41.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836492 2024-09-18T15:47:41.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:47:41.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:47:41.753 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836492 2024-09-18T15:47:41.753 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:47:41.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836492 2024-09-18T15:47:41.756 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836492 2024-09-18T15:47:41.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-09-18T15:47:41.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:47:42.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836492 -lt 21474836492 2024-09-18T15:47:42.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:47:42.106 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672965 2024-09-18T15:47:42.107 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:47:42.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:47:42.111 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672965 2024-09-18T15:47:42.111 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:47:42.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672965 2024-09-18T15:47:42.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-18T15:47:42.113 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672965 2024-09-18T15:47:42.114 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:47:42.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672965 -lt 42949672965 2024-09-18T15:47:42.454 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:47:42.455 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:47:42.455 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:47:42.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T15:47:42.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:47:42.827 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:47:42.827 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:47:42.827 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:47:42.827 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:47:42.827 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:47:42.828 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:47:43.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T15:47:43.179 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:47:43.179 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:47:43.180 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:47:43.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T15:47:43.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:47:43.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:47:43.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5963: TEST_corrupt_snapset_scrub_rep: create_pool foo 1 2024-09-18T15:47:43.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-09-18T15:47:44.073 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'foo' created 2024-09-18T15:47:44.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:47:45.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: create_pool csr_pool 1 1 2024-09-18T15:47:45.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2024-09-18T15:47:45.864 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'csr_pool' created 2024-09-18T15:47:45.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:47:46.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5965: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-09-18T15:47:46.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:47:46.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:47:46.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:47:46.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:47:46.881 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:47:46.882 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:47:46.882 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:47:46.883 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:47:46.883 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:47:46.883 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:47:47.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:47:47.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:47:47.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:47:47.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:47:47.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:47:47.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:47:47.428 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:47:47.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:47:47.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:47:47.429 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:47:47.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836494 2024-09-18T15:47:47.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836494 2024-09-18T15:47:47.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836494' 2024-09-18T15:47:47.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:47:47.564 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:47:47.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672968 2024-09-18T15:47:47.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672968 2024-09-18T15:47:47.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836494 1-42949672968' 2024-09-18T15:47:47.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:47:47.707 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836494 2024-09-18T15:47:47.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:47:47.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:47:47.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836494 2024-09-18T15:47:47.712 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:47:47.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836494 2024-09-18T15:47:47.714 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836494 2024-09-18T15:47:47.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-09-18T15:47:47.715 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:47:48.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836495 -lt 21474836494 2024-09-18T15:47:48.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:47:48.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672968 2024-09-18T15:47:48.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:47:48.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:47:48.035 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672968 2024-09-18T15:47:48.035 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:47:48.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672968 2024-09-18T15:47:48.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-09-18T15:47:48.037 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672968 2024-09-18T15:47:48.037 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:47:48.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672968 -lt 42949672968 2024-09-18T15:47:48.359 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:47:48.359 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:47:48.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:47:48.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:47:48.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:47:48.779 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:47:48.779 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:47:48.779 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:47:48.780 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:47:48.780 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:47:48.780 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:47:49.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:47:49.113 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:47:49.113 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:47:49.114 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:47:49.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:47:49.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:47:49.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:47:49.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5967: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-09-18T15:47:49.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5967: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-09-18T15:47:49.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5968: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-09-18T15:47:49.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5969: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-09-18T15:47:49.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:47:49.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2024-09-18T15:47:49.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ1 2024-09-18T15:47:49.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:47:49.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:47:49.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:47:50.082 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:47:50.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:47:50.634 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:47:50.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:47:50.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:47:50.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:47:50.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5971: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-09-18T15:47:51.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5972: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-09-18T15:47:51.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5967: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-09-18T15:47:51.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5968: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-09-18T15:47:51.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5969: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-09-18T15:47:51.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:47:51.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=csr_pool 2024-09-18T15:47:51.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=ROBJ2 2024-09-18T15:47:51.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:47:51.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:47:51.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:47:51.629 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:47:51.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:47:52.281 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:47:52.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:47:52.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:47:52.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:47:52.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5971: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-09-18T15:47:52.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5972: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-09-18T15:47:52.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5975: TEST_corrupt_snapset_scrub_rep: get_pg csr_pool ROBJ0 2024-09-18T15:47:52.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-09-18T15:47:52.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-09-18T15:47:52.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-09-18T15:47:52.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:47:53.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5975: TEST_corrupt_snapset_scrub_rep: local pg=3.0 2024-09-18T15:47:53.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5976: TEST_corrupt_snapset_scrub_rep: get_primary csr_pool ROBJ0 2024-09-18T15:47:53.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=csr_pool 2024-09-18T15:47:53.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=ROBJ0 2024-09-18T15:47:53.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:47:53.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-09-18T15:47:53.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5976: TEST_corrupt_snapset_scrub_rep: local primary=1 2024-09-18T15:47:53.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5978: TEST_corrupt_snapset_scrub_rep: rados -p csr_pool mksnap snap1 2024-09-18T15:47:53.713 INFO:tasks.workunit.client.0.smithi046.stdout:created pool csr_pool snap snap1 2024-09-18T15:47:53.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5979: TEST_corrupt_snapset_scrub_rep: echo -n head_of_snapshot_data 2024-09-18T15:47:53.719 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5981: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-09-18T15:47:53.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5981: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-09-18T15:47:53.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5982: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-09-18T15:47:53.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: expr 1 % 2 2024-09-18T15:47:53.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: local osd=1 2024-09-18T15:47:53.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5987: TEST_corrupt_snapset_scrub_rep: case $i in 2024-09-18T15:47:53.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5989: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/change 2024-09-18T15:47:53.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5990: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-09-18T15:47:53.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:47:53.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:47:53.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:47:53.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:47:53.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-09-18T15:47:53.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:47:53.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:47:53.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:47:53.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:47:53.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:47:53.998 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:47:53.998 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:47:53.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:47:53.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:47:53.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:47:54.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:47:54.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-09-18T15:47:54.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:47:54.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:47:54.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:47:54.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:47:54.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:47:54.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 --head ROBJ1 clear-snapset corrupt 2024-09-18T15:47:56.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-18T15:47:56.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:47:56.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:47:56.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:47:56.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:47:56.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:47:56.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:47:56.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:47:56.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:47:56.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:47:56.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:47:56.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:47:56.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:47:56.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:47:56.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:47:56.645 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:47:56.645 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:47:56.645 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:47:56.645 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:47:56.645 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:47:56.645 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:47:56.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:47:56.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:47:56.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:47:56.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:47:56.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:47:56.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:47:56.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:47:56.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:47:56.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:47:56.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:47:56.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:47:56.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:47:56.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:47:56.649 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:47:56.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:47:56.650 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:47:56.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:47:56.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:47:56.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:47:56.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:47:56.681 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:47:56.675+0000 7f9fa3705540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:47:56.683 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:47:56.675+0000 7f9fa3705540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:47:56.685 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:47:56.679+0000 7f9fa3705540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:47:56.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:47:56.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:47:56.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:47:56.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:47:56.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:47:56.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:56.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:47:56.970 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:47:56.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:56.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:47:57.230 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:47:57.223+0000 7f9fa3705540 -1 Falling back to public interface 2024-09-18T15:47:57.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:58.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:58.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:58.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:47:58.315 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:47:58.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:58.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:47:58.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:59.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:59.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:59.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:47:59.639 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:47:59.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:59.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:47:59.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:48:00.120 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:00.115+0000 7f9fa3705540 -1 osd.1 31 log_to_monitors true 2024-09-18T15:48:00.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:48:00.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:48:00.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:48:00.947 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:48:00.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:48:00.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:48:01.299 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [10,31) [v2:127.0.0.1:6810/404954,v1:127.0.0.1:6811/404954] [v2:127.0.0.1:6812/404954,v1:127.0.0.1:6813/404954] exists,up 7a06f774-5cc7-45a2-8c6c-0e7441aa66fe 2024-09-18T15:48:01.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:48:01.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:48:01.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:48:01.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:48:01.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:48:01.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:48:01.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:48:01.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:48:01.300 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:48:01.301 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:48:01.301 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:48:01.301 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:48:01.301 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:48:01.301 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:48:01.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:48:01.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:48:01.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:48:01.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:48:01.532 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:48:01.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:48:01.888 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:48:01.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:48:01.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:48:01.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:48:02.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836498 2024-09-18T15:48:02.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836498 2024-09-18T15:48:02.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498' 2024-09-18T15:48:02.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:48:02.025 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:48:02.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855362 2024-09-18T15:48:02.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855362 2024-09-18T15:48:02.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498 1-150323855362' 2024-09-18T15:48:02.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:48:02.187 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836498 2024-09-18T15:48:02.187 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:48:02.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:48:02.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836498 2024-09-18T15:48:02.192 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:48:02.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836498 2024-09-18T15:48:02.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-09-18T15:48:02.193 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836498 2024-09-18T15:48:02.194 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:48:02.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836497 -lt 21474836498 2024-09-18T15:48:02.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:48:03.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:48:03.538 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:48:03.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836497 -lt 21474836498 2024-09-18T15:48:03.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:48:04.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-18T15:48:04.855 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:48:05.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836499 -lt 21474836498 2024-09-18T15:48:05.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:48:05.178 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-150323855362 2024-09-18T15:48:05.179 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:48:05.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:48:05.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-150323855362 2024-09-18T15:48:05.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:48:05.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855362 2024-09-18T15:48:05.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2024-09-18T15:48:05.185 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 150323855362 2024-09-18T15:48:05.185 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:48:05.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855362 -lt 150323855362 2024-09-18T15:48:05.529 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:48:05.530 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:48:05.530 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:48:05.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:48:05.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:48:05.945 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:48:05.945 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:48:05.945 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:48:05.946 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:48:05.946 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:48:05.946 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:48:06.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:48:06.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:48:06.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:48:06.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:48:06.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:48:06.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:48:06.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:48:06.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5981: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-09-18T15:48:06.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5982: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-09-18T15:48:06.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: expr 2 % 2 2024-09-18T15:48:06.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: local osd=0 2024-09-18T15:48:06.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5987: TEST_corrupt_snapset_scrub_rep: case $i in 2024-09-18T15:48:06.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5994: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/change 2024-09-18T15:48:06.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5995: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-09-18T15:48:06.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:48:06.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:48:06.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:48:06.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:48:06.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-09-18T15:48:06.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:48:06.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:48:06.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:48:06.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:48:06.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:48:06.807 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:48:06.807 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:48:06.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:48:06.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:48:06.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:48:07.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:48:07.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-09-18T15:48:07.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:48:07.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:48:07.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:48:07.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:48:07.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:48:07.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 --head ROBJ2 clear-snapset corrupt 2024-09-18T15:48:09.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-18T15:48:09.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:48:09.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:48:09.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:48:09.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:48:09.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:48:09.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:48:09.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:48:09.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:48:09.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:48:09.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:48:09.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:48:09.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:48:09.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:48:09.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:48:09.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:48:09.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:48:09.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:48:09.499 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:48:09.499 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:09.499 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:48:09.499 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:48:09.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:48:09.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:48:09.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:48:09.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:48:09.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:48:09.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:48:09.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:48:09.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:48:09.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:48:09.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:48:09.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:48:09.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:48:09.502 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:48:09.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:48:09.502 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:48:09.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:48:09.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:48:09.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:48:09.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:48:09.532 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:09.527+0000 7f548c4df540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:09.536 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:09.531+0000 7f548c4df540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:09.538 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:09.531+0000 7f548c4df540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:09.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:48:09.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:48:09.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:48:09.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:48:09.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:48:09.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:48:09.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:48:09.809 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:48:09.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:48:09.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:48:10.091 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:10.083+0000 7f548c4df540 -1 Falling back to public interface 2024-09-18T15:48:10.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:48:11.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:48:11.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:48:11.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:48:11.190 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:48:11.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:48:11.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:48:11.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:48:12.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:48:12.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:48:12.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:48:12.509 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:48:12.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:48:12.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:48:12.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:48:13.070 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:13.063+0000 7f548c4df540 -1 osd.0 37 log_to_monitors true 2024-09-18T15:48:13.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:48:13.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:48:13.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:48:13.859 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:48:13.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:48:13.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:48:14.334 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 41 up_thru 41 down_at 38 last_clean_interval [5,37) [v2:127.0.0.1:6802/406301,v1:127.0.0.1:6803/406301] [v2:127.0.0.1:6804/406301,v1:127.0.0.1:6805/406301] exists,up f5b28f60-d48d-403c-a22a-a35a92723ee3 2024-09-18T15:48:14.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:48:14.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:48:14.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:48:14.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:48:14.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:48:14.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:48:14.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:48:14.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:48:14.335 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:48:14.336 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:48:14.336 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:48:14.336 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:48:14.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:48:14.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:48:14.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:48:14.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:48:14.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:48:14.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:48:14.571 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:48:14.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:48:14.882 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:48:14.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:48:14.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:48:14.883 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:48:15.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=176093659138 2024-09-18T15:48:15.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 176093659138 2024-09-18T15:48:15.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-176093659138' 2024-09-18T15:48:15.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:48:15.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:48:15.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855365 2024-09-18T15:48:15.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855365 2024-09-18T15:48:15.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-176093659138 1-150323855365' 2024-09-18T15:48:15.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:48:15.212 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-176093659138 2024-09-18T15:48:15.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:48:15.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:48:15.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-176093659138 2024-09-18T15:48:15.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:48:15.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=176093659138 2024-09-18T15:48:15.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 176093659138' 2024-09-18T15:48:15.220 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 176093659138 2024-09-18T15:48:15.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:48:15.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 176093659138 2024-09-18T15:48:15.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:48:16.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:48:16.530 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:48:16.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 176093659138 -lt 176093659138 2024-09-18T15:48:16.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:48:16.840 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-150323855365 2024-09-18T15:48:16.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:48:16.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:48:16.844 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-150323855365 2024-09-18T15:48:16.845 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:48:16.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855365 2024-09-18T15:48:16.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 150323855365' 2024-09-18T15:48:16.847 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 150323855365 2024-09-18T15:48:16.848 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:48:17.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855366 -lt 150323855365 2024-09-18T15:48:17.185 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:48:17.186 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:48:17.186 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:48:17.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T15:48:17.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:48:17.594 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:48:17.594 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:48:17.594 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:48:17.594 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:48:17.594 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:48:17.595 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:48:17.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T15:48:17.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:48:17.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:48:17.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:48:18.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T15:48:18.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:48:18.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:48:18.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6000: TEST_corrupt_snapset_scrub_rep: rm td/osd-scrub-repair/change 2024-09-18T15:48:18.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6002: TEST_corrupt_snapset_scrub_rep: pg_scrub 3.0 2024-09-18T15:48:18.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: pg_scrub: local pgid=3.0 2024-09-18T15:48:18.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: pg_scrub: get_last_scrub_stamp 3.0 2024-09-18T15:48:18.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:48:18.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:48:18.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:48:18.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:48:18.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: pg_scrub: local last_scrub=2024-09-18T15:47:45.556952+0000 2024-09-18T15:48:18.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_scrub: ceph pg scrub 3.0 2024-09-18T15:48:18.907 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 3.0 on osd.1 to scrub 2024-09-18T15:48:18.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_scrub: wait_for_scrub 3.0 2024-09-18T15:47:45.556952+0000 2024-09-18T15:48:18.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=3.0 2024-09-18T15:48:18.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:47:45.556952+0000 2024-09-18T15:48:18.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:48:18.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:48:18.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:48:18.917 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:48:18.917 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:48:18.917 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:48:18.918 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:48:18.918 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:48:19.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:47:45.556952+0000 '>' 2024-09-18T15:47:45.556952+0000 2024-09-18T15:48:19.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:48:20.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:48:20.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:48:20.215 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:48:20.215 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:48:20.215 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:48:20.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:48:20.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:48:20.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:47:45.556952+0000 '>' 2024-09-18T15:47:45.556952+0000 2024-09-18T15:48:20.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:48:21.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:48:21.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:48:21.536 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-18T15:48:21.536 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-09-18T15:48:21.536 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:48:21.537 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:48:21.537 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-18T15:48:21.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:48:19.016901+0000 '>' 2024-09-18T15:47:45.556952+0000 2024-09-18T15:48:21.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:48:21.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6004: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-pg csr_pool 2024-09-18T15:48:21.914 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6006: TEST_corrupt_snapset_scrub_rep: jq '. | length' td/osd-scrub-repair/json 2024-09-18T15:48:21.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6006: TEST_corrupt_snapset_scrub_rep: test 1 = 1 2024-09-18T15:48:21.958 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6008: TEST_corrupt_snapset_scrub_rep: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-18T15:48:22.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6008: TEST_corrupt_snapset_scrub_rep: test 3.0 = 3.0 2024-09-18T15:48:22.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6010: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-obj 3.0 2024-09-18T15:48:22.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6012: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-09-18T15:48:22.065 INFO:tasks.workunit.client.0.smithi046.stderr: . as $in 2024-09-18T15:48:22.066 INFO:tasks.workunit.client.0.smithi046.stderr: | if type == "object" then 2024-09-18T15:48:22.066 INFO:tasks.workunit.client.0.smithi046.stderr: reduce keys[] as $key 2024-09-18T15:48:22.066 INFO:tasks.workunit.client.0.smithi046.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-18T15:48:22.066 INFO:tasks.workunit.client.0.smithi046.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-18T15:48:22.066 INFO:tasks.workunit.client.0.smithi046.stderr: else f 2024-09-18T15:48:22.066 INFO:tasks.workunit.client.0.smithi046.stderr: end; 2024-09-18T15:48:22.066 INFO:tasks.workunit.client.0.smithi046.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-18T15:48:22.066 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-18T15:48:22.066 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-18T15:48:22.066 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-18T15:48:22.066 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-18T15:48:22.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6012: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-09-18T15:48:22.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6012: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-18T15:48:22.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-09-18T15:48:22.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-18T15:48:22.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-09-18T15:48:22.133 INFO:tasks.workunit.client.0.smithi046.stderr: . as $in 2024-09-18T15:48:22.134 INFO:tasks.workunit.client.0.smithi046.stderr: | if type == "object" then 2024-09-18T15:48:22.134 INFO:tasks.workunit.client.0.smithi046.stderr: reduce keys[] as $key 2024-09-18T15:48:22.134 INFO:tasks.workunit.client.0.smithi046.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-18T15:48:22.134 INFO:tasks.workunit.client.0.smithi046.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-18T15:48:22.134 INFO:tasks.workunit.client.0.smithi046.stderr: else f 2024-09-18T15:48:22.134 INFO:tasks.workunit.client.0.smithi046.stderr: end; 2024-09-18T15:48:22.134 INFO:tasks.workunit.client.0.smithi046.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-18T15:48:22.134 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-18T15:48:22.134 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-18T15:48:22.134 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-18T15:48:22.134 INFO:tasks.workunit.client.0.smithi046.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-18T15:48:22.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6179: TEST_corrupt_snapset_scrub_rep: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-18T15:48:22.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-18T15:48:22.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: test no = yes 2024-09-18T15:48:22.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6185: TEST_corrupt_snapset_scrub_rep: test '' = yes 2024-09-18T15:48:22.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6190: TEST_corrupt_snapset_scrub_rep: ERRORS=0 2024-09-18T15:48:22.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6191: TEST_corrupt_snapset_scrub_rep: declare -a err_strings 2024-09-18T15:48:22.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6192: TEST_corrupt_snapset_scrub_rep: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' 2024-09-18T15:48:22.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6193: TEST_corrupt_snapset_scrub_rep: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' 2024-09-18T15:48:22.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6194: TEST_corrupt_snapset_scrub_rep: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' 2024-09-18T15:48:22.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6195: TEST_corrupt_snapset_scrub_rep: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-09-18T15:48:22.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6196: TEST_corrupt_snapset_scrub_rep: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' 2024-09-18T15:48:22.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6197: TEST_corrupt_snapset_scrub_rep: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' 2024-09-18T15:48:22.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6199: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-09-18T15:48:22.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6201: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-09-18T15:48:22.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6199: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-09-18T15:48:22.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6201: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-09-18T15:48:22.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6199: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-09-18T15:48:22.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6201: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' td/osd-scrub-repair/osd.1.log 2024-09-18T15:48:22.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6199: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-09-18T15:48:22.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6201: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-09-18T15:48:22.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6199: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-09-18T15:48:22.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6201: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-09-18T15:48:22.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6199: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-09-18T15:48:22.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6201: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' td/osd-scrub-repair/osd.1.log 2024-09-18T15:48:22.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6208: TEST_corrupt_snapset_scrub_rep: '[' 0 '!=' 0 ']' 2024-09-18T15:48:22.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6214: TEST_corrupt_snapset_scrub_rep: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-09-18T15:48:22.782 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'csr_pool' removed 2024-09-18T15:48:22.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T15:48:22.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:48:22.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:48:22.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:48:22.798 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:48:22.798 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:48:22.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:48:22.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:48:22.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:48:22.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:48:22.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:48:22.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:48:22.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:48:22.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:48:22.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:48:22.926 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:48:22.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:48:22.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:48:22.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:48:22.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:48:22.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:48:22.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:48:22.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:48:22.954 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:48:22.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:22.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:48:22.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:48:22.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:48:22.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:48:22.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T15:48:22.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T15:48:22.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T15:48:22.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T15:48:22.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:48:22.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:48:22.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:48:22.959 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:48:22.959 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:48:22.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:48:22.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:48:22.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:48:22.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:48:22.965 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:48:22.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:48:22.967 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:48:22.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:48:22.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:48:22.970 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:48:22.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:48:22.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:48:22.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:48:22.973 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:48:22.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:48:22.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:48:22.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:48:22.980 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:48:22.980 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:22.980 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:48:22.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:48:22.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:48:22.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:48:22.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T15:48:22.986 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:48:22.986 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:22.986 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:48:22.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T15:48:22.989 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:48:22.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:48:22.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:48:22.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T15:48:22.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_list_missing_erasure_coded_appends td/osd-scrub-repair 2024-09-18T15:48:22.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: TEST_list_missing_erasure_coded_appends: list_missing_erasure_coded td/osd-scrub-repair false 2024-09-18T15:48:22.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:929: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-09-18T15:48:22.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:930: list_missing_erasure_coded: local allow_overwrites=false 2024-09-18T15:48:22.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: local poolname=ecpool 2024-09-18T15:48:22.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:933: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-09-18T15:48:22.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T15:48:22.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:48:22.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:48:22.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:48:22.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T15:48:22.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-18T15:48:23.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:48:23.369 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:48:23.369 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:48:23.369 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:48:23.369 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:23.369 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:48:23.370 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:48:23.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T15:48:23.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:48:23.651 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:48:23.651 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:48:23.651 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:48:23.651 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:48:23.652 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:48:23.653 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:48:23.653 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:48:23.653 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:48:23.654 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:23.654 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:23.654 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:48:23.654 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:48:23.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:48:23.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T15:48:23.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:48:23.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:48:23.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:48:23.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:48:23.753 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:48:23.753 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:48:23.753 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:48:23.753 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:48:23.753 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:23.753 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:23.753 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:48:23.753 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:48:23.754 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:48:23.754 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T15:48:23.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:934: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-09-18T15:48:23.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T15:48:23.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:48:23.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:48:23.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:48:23.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T15:48:23.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:48:24.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:48:24.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:48:24.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:48:24.700 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:48:24.701 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:24.701 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:48:24.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:48:24.702 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:48:24.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:48:24.740 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: seq 0 2 2024-09-18T15:48:24.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-18T15:48:24.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: '[' false = true ']' 2024-09-18T15:48:24.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: run_osd_filestore td/osd-scrub-repair 0 2024-09-18T15:48:24.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T15:48:24.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T15:48:24.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=0 2024-09-18T15:48:24.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T15:48:24.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:48:24.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:48:24.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:48:24.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:48:24.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:48:24.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:48:24.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:48:24.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T15:48:24.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T15:48:24.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:48:24.746 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T15:48:24.746 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:48:24.746 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:48:24.746 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:48:24.746 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:24.746 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:48:24.747 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:48:24.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:48:24.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T15:48:24.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T15:48:24.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T15:48:24.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:48:24.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:48:24.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:48:24.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:48:24.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:48:24.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T15:48:24.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T15:48:24.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:48:24.748 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T15:48:24.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=4c715e51-d1bd-4dcf-b102-acaac7c3f20c 2024-09-18T15:48:24.749 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 4c715e51-d1bd-4dcf-b102-acaac7c3f20c 2024-09-18T15:48:24.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd1 4c715e51-d1bd-4dcf-b102-acaac7c3f20c' 2024-09-18T15:48:24.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T15:48:24.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQBI9upmwHq3LRAAdOj8gruBNlcaJUy8VS9t9g== 2024-09-18T15:48:24.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQBI9upmwHq3LRAAdOj8gruBNlcaJUy8VS9t9g=="}' 2024-09-18T15:48:24.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 4c715e51-d1bd-4dcf-b102-acaac7c3f20c -i td/osd-scrub-repair/0/new.json 2024-09-18T15:48:25.051 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:48:25.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/0/new.json 2024-09-18T15:48:25.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBI9upmwHq3LRAAdOj8gruBNlcaJUy8VS9t9g== --osd-uuid 4c715e51-d1bd-4dcf-b102-acaac7c3f20c --osd-objectstore=filestore 2024-09-18T15:48:25.100 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:25.091+0000 7f72e446d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:25.143 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:25.135+0000 7f72e446d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:25.145 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:25.139+0000 7f72e446d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:25.561 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:25.555+0000 7f72e446d540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:48:26.040 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:26.031+0000 7f72e446d540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:48:26.041 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:26.035+0000 7f72e446d540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:48:26.041 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:26.035+0000 7f72e446d540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:48:26.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T15:48:26.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T15:48:26.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd0 key to auth repository 2024-09-18T15:48:26.617 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T15:48:26.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:48:26.799 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: key for osd.0 not found in provided keyring 2024-09-18T15:48:26.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.0 2024-09-18T15:48:26.801 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T15:48:26.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:48:26.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T15:48:26.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T15:48:26.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T15:48:26.825 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:26.819+0000 7f4db74f5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:26.827 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:26.823+0000 7f4db74f5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:26.831 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:26.823+0000 7f4db74f5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:26.840 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:26.835+0000 7f4db74f5540 -1 Falling back to public interface 2024-09-18T15:48:27.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 0 2024-09-18T15:48:27.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:48:27.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:48:27.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:48:27.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:48:27.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:48:27.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:48:27.001 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:48:27.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:48:27.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:48:27.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:48:27.413 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:27.407+0000 7f4db74f5540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:48:27.420 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:27.415+0000 7f4db74f5540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:48:27.421 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:27.415+0000 7f4db74f5540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:48:27.429 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:27.423+0000 7f4db74f5540 -1 osd.0 0 log_to_monitors true 2024-09-18T15:48:28.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:48:28.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:48:28.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:48:28.176 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:48:28.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:48:28.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:48:28.357 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/407729,v1:127.0.0.1:6801/407729] [v2:127.0.0.1:6802/407729,v1:127.0.0.1:6803/407729] exists,up 4c715e51-d1bd-4dcf-b102-acaac7c3f20c 2024-09-18T15:48:28.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:48:28.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:48:28.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:48:28.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-18T15:48:28.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: '[' false = true ']' 2024-09-18T15:48:28.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: run_osd_filestore td/osd-scrub-repair 1 2024-09-18T15:48:28.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T15:48:28.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T15:48:28.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=1 2024-09-18T15:48:28.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T15:48:28.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:48:28.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:48:28.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:48:28.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:48:28.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:48:28.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:48:28.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:48:28.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T15:48:28.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T15:48:28.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:48:28.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T15:48:28.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:48:28.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:48:28.361 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:48:28.361 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:28.361 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:48:28.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:48:28.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:48:28.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T15:48:28.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T15:48:28.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T15:48:28.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:48:28.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:48:28.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:48:28.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:48:28.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:48:28.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T15:48:28.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T15:48:28.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:48:28.364 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T15:48:28.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=3c22d1f4-e86b-49df-ac83-08388d0628e8 2024-09-18T15:48:28.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd1 3c22d1f4-e86b-49df-ac83-08388d0628e8' 2024-09-18T15:48:28.365 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 3c22d1f4-e86b-49df-ac83-08388d0628e8 2024-09-18T15:48:28.365 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T15:48:28.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQBM9upmKK7mFhAA/Yqs4vu+0YBRRlExzUOEFQ== 2024-09-18T15:48:28.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQBM9upmKK7mFhAA/Yqs4vu+0YBRRlExzUOEFQ=="}' 2024-09-18T15:48:28.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 3c22d1f4-e86b-49df-ac83-08388d0628e8 -i td/osd-scrub-repair/1/new.json 2024-09-18T15:48:28.779 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:48:28.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/1/new.json 2024-09-18T15:48:28.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBM9upmKK7mFhAA/Yqs4vu+0YBRRlExzUOEFQ== --osd-uuid 3c22d1f4-e86b-49df-ac83-08388d0628e8 --osd-objectstore=filestore 2024-09-18T15:48:28.826 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:28.819+0000 7f4bf072e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:28.829 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:28.823+0000 7f4bf072e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:28.831 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:28.823+0000 7f4bf072e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:29.364 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:29.359+0000 7f4bf072e540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:48:29.827 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:29.819+0000 7f4bf072e540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:48:29.827 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:29.819+0000 7f4bf072e540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:48:29.827 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:29.819+0000 7f4bf072e540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:48:30.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-18T15:48:30.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T15:48:30.703 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T15:48:30.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd1 key to auth repository 2024-09-18T15:48:30.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:48:31.107 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T15:48:31.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.1 2024-09-18T15:48:31.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T15:48:31.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T15:48:31.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T15:48:31.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:48:31.138 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:31.131+0000 7fb3b6a15540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:31.140 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:31.135+0000 7fb3b6a15540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:31.142 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:31.135+0000 7fb3b6a15540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:31.144 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:31.139+0000 7fb3b6a15540 -1 Falling back to public interface 2024-09-18T15:48:31.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 1 2024-09-18T15:48:31.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:48:31.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:48:31.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:48:31.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:48:31.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:48:31.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:48:31.472 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:48:31.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:48:31.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:48:31.639 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:31.631+0000 7fb3b6a15540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:48:31.639 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:31.631+0000 7fb3b6a15540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:48:31.639 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:31.631+0000 7fb3b6a15540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:48:31.649 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:31.643+0000 7fb3b6a15540 -1 osd.1 0 log_to_monitors true 2024-09-18T15:48:31.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:48:32.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:48:32.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:48:32.843 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:48:32.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:48:32.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:48:32.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:48:33.203 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/408113,v1:127.0.0.1:6811/408113] [v2:127.0.0.1:6812/408113,v1:127.0.0.1:6813/408113] exists,up 3c22d1f4-e86b-49df-ac83-08388d0628e8 2024-09-18T15:48:33.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:48:33.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:48:33.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:48:33.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-18T15:48:33.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: '[' false = true ']' 2024-09-18T15:48:33.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: run_osd_filestore td/osd-scrub-repair 2 2024-09-18T15:48:33.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T15:48:33.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T15:48:33.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=2 2024-09-18T15:48:33.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T15:48:33.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:48:33.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:48:33.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:48:33.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:48:33.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:48:33.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T15:48:33.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T15:48:33.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T15:48:33.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T15:48:33.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:48:33.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T15:48:33.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:48:33.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:48:33.209 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:48:33.209 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:33.209 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:48:33.209 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:48:33.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:48:33.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T15:48:33.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T15:48:33.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T15:48:33.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:48:33.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:48:33.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:48:33.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:48:33.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:48:33.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T15:48:33.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T15:48:33.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/2 2024-09-18T15:48:33.210 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T15:48:33.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=94b83675-9900-41b4-8639-78375059a3b0 2024-09-18T15:48:33.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd1 94b83675-9900-41b4-8639-78375059a3b0' 2024-09-18T15:48:33.213 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 94b83675-9900-41b4-8639-78375059a3b0 2024-09-18T15:48:33.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T15:48:33.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQBR9upmGnqKDRAAhNEpZbpOGUJwbxG9fRhILw== 2024-09-18T15:48:33.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQBR9upmGnqKDRAAhNEpZbpOGUJwbxG9fRhILw=="}' 2024-09-18T15:48:33.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 94b83675-9900-41b4-8639-78375059a3b0 -i td/osd-scrub-repair/2/new.json 2024-09-18T15:48:33.623 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:48:33.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/2/new.json 2024-09-18T15:48:33.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBR9upmGnqKDRAAhNEpZbpOGUJwbxG9fRhILw== --osd-uuid 94b83675-9900-41b4-8639-78375059a3b0 --osd-objectstore=filestore 2024-09-18T15:48:33.671 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:33.663+0000 7fd8c8f8b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:33.675 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:33.667+0000 7fd8c8f8b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:33.677 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:33.671+0000 7fd8c8f8b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:34.384 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:34.379+0000 7fd8c8f8b540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:48:35.198 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:35.191+0000 7fd8c8f8b540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:48:35.198 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:35.191+0000 7fd8c8f8b540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:48:35.198 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:35.191+0000 7fd8c8f8b540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:48:35.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-18T15:48:35.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T15:48:35.813 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T15:48:35.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd2 key to auth repository 2024-09-18T15:48:35.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:48:36.186 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: key for osd.2 not found in provided keyring 2024-09-18T15:48:36.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.2 2024-09-18T15:48:36.190 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T15:48:36.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:48:36.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T15:48:36.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T15:48:36.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T15:48:36.216 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:36.211+0000 7f4b745a2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:36.218 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:36.211+0000 7f4b745a2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:36.220 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:36.215+0000 7f4b745a2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:36.223 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:36.219+0000 7f4b745a2540 -1 Falling back to public interface 2024-09-18T15:48:36.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 2 2024-09-18T15:48:36.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:48:36.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:48:36.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:48:36.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:48:36.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:48:36.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:48:36.565 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:48:36.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:48:36.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:48:36.968 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:36.963+0000 7f4b745a2540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:48:36.971 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:36.963+0000 7f4b745a2540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:48:36.971 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:36.963+0000 7f4b745a2540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:48:36.981 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:48:36.975+0000 7f4b745a2540 -1 osd.2 0 log_to_monitors true 2024-09-18T15:48:37.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:48:38.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:48:38.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:48:38.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:48:38.034 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:48:38.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:48:38.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:48:38.360 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/408396,v1:127.0.0.1:6819/408396] [v2:127.0.0.1:6820/408396,v1:127.0.0.1:6821/408396] exists,up 94b83675-9900-41b4-8639-78375059a3b0 2024-09-18T15:48:38.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:48:38.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:48:38.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:48:38.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:942: list_missing_erasure_coded: create_rbd_pool 2024-09-18T15:48:38.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T15:48:38.664 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' does not exist 2024-09-18T15:48:38.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T15:48:38.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T15:48:39.174 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' created 2024-09-18T15:48:39.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:48:40.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T15:48:42.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: wait_for_clean 2024-09-18T15:48:42.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:48:42.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:48:42.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:48:42.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:48:42.266 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:48:42.267 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:48:42.267 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:48:42.268 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:48:42.268 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:48:42.268 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:48:42.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:48:42.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:48:42.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:48:42.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:48:42.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:48:42.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:48:42.844 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:48:42.844 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:48:42.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:48:42.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:48:42.844 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:48:42.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836484 2024-09-18T15:48:42.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836484 2024-09-18T15:48:42.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836484' 2024-09-18T15:48:42.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:48:42.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:48:43.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672964 2024-09-18T15:48:43.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672964 2024-09-18T15:48:43.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836484 1-42949672964' 2024-09-18T15:48:43.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:48:43.272 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:48:43.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509442 2024-09-18T15:48:43.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509442 2024-09-18T15:48:43.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836484 1-42949672964 2-64424509442' 2024-09-18T15:48:43.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:48:43.419 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836484 2024-09-18T15:48:43.419 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:48:43.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:48:43.422 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836484 2024-09-18T15:48:43.423 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:48:43.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836484 2024-09-18T15:48:43.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2024-09-18T15:48:43.425 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836484 2024-09-18T15:48:43.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:48:43.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836483 -lt 21474836484 2024-09-18T15:48:43.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:48:44.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:48:44.734 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:48:45.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836484 -lt 21474836484 2024-09-18T15:48:45.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:48:45.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672964 2024-09-18T15:48:45.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:48:45.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:48:45.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672964 2024-09-18T15:48:45.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:48:45.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672964 2024-09-18T15:48:45.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-09-18T15:48:45.059 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672964 2024-09-18T15:48:45.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:48:45.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672964 -lt 42949672964 2024-09-18T15:48:45.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:48:45.400 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509442 2024-09-18T15:48:45.401 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:48:45.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:48:45.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509442 2024-09-18T15:48:45.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:48:45.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509442 2024-09-18T15:48:45.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-09-18T15:48:45.408 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509442 2024-09-18T15:48:45.409 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:48:45.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509443 -lt 64424509442 2024-09-18T15:48:45.745 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:48:45.745 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:48:45.745 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:48:46.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T15:48:46.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:48:46.150 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:48:46.150 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:48:46.150 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:48:46.150 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:48:46.150 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:48:46.150 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:48:46.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T15:48:46.490 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:48:46.491 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:48:46.491 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:48:46.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T15:48:46.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:48:46.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:48:46.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:945: list_missing_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-09-18T15:48:46.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: create_ec_pool: local pool_name=ecpool 2024-09-18T15:48:46.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: create_ec_pool: shift 2024-09-18T15:48:46.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: create_ec_pool: local allow_overwrites=false 2024-09-18T15:48:46.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: create_ec_pool: shift 2024-09-18T15:48:46.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-18T15:48:47.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-18T15:48:47.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-18T15:48:48.121 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'ecpool' created 2024-09-18T15:48:48.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:48:49.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: create_ec_pool: '[' false = true ']' 2024-09-18T15:48:49.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: create_ec_pool: wait_for_clean 2024-09-18T15:48:49.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:48:49.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:48:49.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:48:49.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:48:49.137 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:48:49.138 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:48:49.138 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:48:49.139 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:48:49.139 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:48:49.139 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:48:49.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:48:49.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:48:49.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:48:49.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:48:49.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:48:49.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:48:49.682 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:48:49.682 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:48:49.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:48:49.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:48:49.682 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:48:49.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836486 2024-09-18T15:48:49.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836486 2024-09-18T15:48:49.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836486' 2024-09-18T15:48:49.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:48:49.866 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:48:49.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672966 2024-09-18T15:48:49.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672966 2024-09-18T15:48:49.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836486 1-42949672966' 2024-09-18T15:48:49.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:48:49.998 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:48:50.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509445 2024-09-18T15:48:50.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509445 2024-09-18T15:48:50.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836486 1-42949672966 2-64424509445' 2024-09-18T15:48:50.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:48:50.204 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836486 2024-09-18T15:48:50.204 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:48:50.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:48:50.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836486 2024-09-18T15:48:50.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:48:50.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836486 2024-09-18T15:48:50.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-09-18T15:48:50.211 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836486 2024-09-18T15:48:50.211 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:48:50.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836486 -lt 21474836486 2024-09-18T15:48:50.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:48:50.515 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672966 2024-09-18T15:48:50.516 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:48:50.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:48:50.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672966 2024-09-18T15:48:50.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:48:50.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672966 2024-09-18T15:48:50.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-09-18T15:48:50.522 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672966 2024-09-18T15:48:50.523 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:48:50.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672966 -lt 42949672966 2024-09-18T15:48:50.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:48:50.848 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509445 2024-09-18T15:48:50.848 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:48:50.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:48:50.851 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509445 2024-09-18T15:48:50.851 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:48:50.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509445 2024-09-18T15:48:50.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-09-18T15:48:50.853 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509445 2024-09-18T15:48:50.854 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:48:51.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509444 -lt 64424509445 2024-09-18T15:48:51.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:48:52.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:48:52.178 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:48:52.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509445 -lt 64424509445 2024-09-18T15:48:52.494 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:48:52.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:48:52.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:48:52.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:48:52.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:48:52.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:48:52.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:48:52.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:48:52.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:48:52.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:48:52.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:48:53.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:48:53.228 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:48:53.229 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:48:53.229 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:48:53.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:48:53.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:48:53.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:48:53.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: create_ec_pool: return 0 2024-09-18T15:48:53.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:948: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-09-18T15:48:53.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:48:53.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2024-09-18T15:48:53.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=MOBJ0 2024-09-18T15:48:53.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:48:53.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:48:53.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:48:54.207 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:48:54.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:48:54.735 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:48:54.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:48:54.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:48:54.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:48:54.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:949: list_missing_erasure_coded: osds0=($(get_osds $poolname MOBJ0)) 2024-09-18T15:48:54.890 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:949: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-09-18T15:48:54.891 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-09-18T15:48:54.891 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ0 2024-09-18T15:48:54.892 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ0 2024-09-18T15:48:54.892 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:48:55.203 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-18T15:48:55.203 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:48:55.203 INFO:tasks.workunit.client.0.smithi046.stderr:0' 2024-09-18T15:48:55.203 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-09-18T15:48:55.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:949: list_missing_erasure_coded: local -a osds0 2024-09-18T15:48:55.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:952: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-09-18T15:48:55.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:48:55.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2024-09-18T15:48:55.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=MOBJ1 2024-09-18T15:48:55.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:48:55.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:48:55.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:48:55.695 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:48:55.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:48:56.174 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:48:56.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:48:56.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:48:56.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:48:56.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: osds1=($(get_osds $poolname MOBJ1)) 2024-09-18T15:48:56.338 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-09-18T15:48:56.339 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-09-18T15:48:56.339 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ1 2024-09-18T15:48:56.340 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ1 2024-09-18T15:48:56.340 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:48:56.681 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-18T15:48:56.681 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:48:56.681 INFO:tasks.workunit.client.0.smithi046.stderr:0' 2024-09-18T15:48:56.681 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-09-18T15:48:56.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: local -a osds1 2024-09-18T15:48:56.682 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:956: list_missing_erasure_coded: seq 0 2 2024-09-18T15:48:56.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:956: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-18T15:48:56.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:48:56.685 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:48:56.685 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:48:56.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:48:56.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:48:56.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:48:57.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:48:57.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:956: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-18T15:48:57.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:48:57.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:48:57.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:48:57.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:48:57.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:48:57.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:48:58.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:48:58.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:956: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-18T15:48:58.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-18T15:48:58.323 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:48:58.324 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:48:58.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:48:58.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:48:58.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:48:58.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:48:58.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: id=2 2024-09-18T15:48:58.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-09-18T15:48:59.147 INFO:tasks.workunit.client.0.smithi046.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-09-18T15:48:59.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:963: list_missing_erasure_coded: id=1 2024-09-18T15:48:59.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-09-18T15:48:59.691 INFO:tasks.workunit.client.0.smithi046.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-09-18T15:48:59.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:967: list_missing_erasure_coded: id=1 2024-09-18T15:48:59.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-09-18T15:49:00.377 INFO:tasks.workunit.client.0.smithi046.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-09-18T15:49:00.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: id=0 2024-09-18T15:49:00.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-09-18T15:49:01.085 INFO:tasks.workunit.client.0.smithi046.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-09-18T15:49:01.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:974: list_missing_erasure_coded: seq 0 2 2024-09-18T15:49:01.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:974: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-18T15:49:01.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:975: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-09-18T15:49:01.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:49:01.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:49:01.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:49:01.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:49:01.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:49:01.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:49:01.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:49:01.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:49:01.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:49:01.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:49:01.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:49:01.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:49:01.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:49:01.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:49:01.331 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:49:01.331 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:49:01.331 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:49:01.332 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:49:01.332 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:49:01.332 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:49:01.332 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:49:01.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:49:01.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:49:01.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:49:01.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:49:01.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:49:01.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:49:01.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:49:01.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:49:01.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:49:01.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:49:01.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:49:01.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:49:01.335 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:49:01.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:49:01.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:49:01.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:49:01.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:49:01.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:49:01.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:49:01.367 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:01.359+0000 7fde325b2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:01.371 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:01.363+0000 7fde325b2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:01.373 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:01.367+0000 7fde325b2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:01.376 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:01.371+0000 7fde325b2540 -1 Falling back to public interface 2024-09-18T15:49:01.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:49:01.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:49:01.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:49:01.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:49:01.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:49:01.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:01.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:49:01.648 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:49:01.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:01.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:01.668 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:01.663+0000 7fde325b2540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:49:01.668 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:01.663+0000 7fde325b2540 -1 journal do_read_entry(401408): bad header magic 2024-09-18T15:49:01.668 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:01.663+0000 7fde325b2540 -1 journal do_read_entry(401408): bad header magic 2024-09-18T15:49:01.678 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:01.671+0000 7fde325b2540 -1 osd.0 34 log_to_monitors true 2024-09-18T15:49:01.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:02.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:02.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:02.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:49:02.957 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:49:02.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:02.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:03.272 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 41 up_thru 41 down_at 35 last_clean_interval [5,34) [v2:127.0.0.1:6800/409839,v1:127.0.0.1:6801/409839] [v2:127.0.0.1:6802/409839,v1:127.0.0.1:6803/409839] exists,up 4c715e51-d1bd-4dcf-b102-acaac7c3f20c 2024-09-18T15:49:03.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:49:03.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:49:03.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:49:03.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:974: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-18T15:49:03.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:975: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-09-18T15:49:03.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:49:03.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:49:03.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:49:03.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:49:03.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:49:03.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:49:03.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:49:03.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:49:03.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:49:03.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:49:03.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:49:03.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:49:03.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:49:03.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:49:03.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:49:03.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:49:03.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:49:03.275 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:49:03.275 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:49:03.275 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:49:03.276 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:49:03.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:49:03.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:49:03.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:49:03.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:49:03.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:49:03.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:49:03.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:49:03.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:49:03.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:49:03.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:49:03.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:49:03.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:49:03.278 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:49:03.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:49:03.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:49:03.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:49:03.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:49:03.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:49:03.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:49:03.308 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:03.303+0000 7fad2f36d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:03.340 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:03.335+0000 7fad2f36d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:03.341 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:03.335+0000 7fad2f36d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:03.345 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:03.339+0000 7fad2f36d540 -1 Falling back to public interface 2024-09-18T15:49:03.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:49:03.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:49:03.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:49:03.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:49:03.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:49:03.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:03.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:49:03.773 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:49:03.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:03.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:49:03.777 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:03.771+0000 7fad2f36d540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:49:03.777 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:03.771+0000 7fad2f36d540 -1 journal do_read_entry(442368): bad header magic 2024-09-18T15:49:03.777 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:03.771+0000 7fad2f36d540 -1 journal do_read_entry(442368): bad header magic 2024-09-18T15:49:03.793 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:03.787+0000 7fad2f36d540 -1 osd.1 37 log_to_monitors true 2024-09-18T15:49:04.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:05.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:05.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:05.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:49:05.117 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:49:05.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:05.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:49:05.433 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 37 last_clean_interval [10,37) [v2:127.0.0.1:6810/410007,v1:127.0.0.1:6811/410007] [v2:127.0.0.1:6812/410007,v1:127.0.0.1:6813/410007] exists,up 3c22d1f4-e86b-49df-ac83-08388d0628e8 2024-09-18T15:49:05.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:49:05.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:49:05.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:49:05.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:974: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-18T15:49:05.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:975: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-09-18T15:49:05.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:49:05.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:49:05.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T15:49:05.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:49:05.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:49:05.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:49:05.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:49:05.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:49:05.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:49:05.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T15:49:05.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T15:49:05.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:49:05.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:49:05.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:49:05.437 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:49:05.437 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:49:05.437 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:49:05.437 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:49:05.437 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:49:05.437 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:49:05.437 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:49:05.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:49:05.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:49:05.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:49:05.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:49:05.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:49:05.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:49:05.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:49:05.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:49:05.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:49:05.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:49:05.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-18T15:49:05.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T15:49:05.439 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.2 2024-09-18T15:49:05.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:49:05.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-18T15:49:05.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T15:49:05.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:49:05.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:49:05.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:49:05.467 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:05.459+0000 7fcfc4215540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:05.492 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:05.487+0000 7fcfc4215540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:05.493 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:05.487+0000 7fcfc4215540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:05.495 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:05.491+0000 7fcfc4215540 -1 Falling back to public interface 2024-09-18T15:49:05.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T15:49:05.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:49:05.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:49:05.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:49:05.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:49:05.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:05.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:49:05.958 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:49:05.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:49:05.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:05.960 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:05.955+0000 7fcfc4215540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:49:05.965 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:05.959+0000 7fcfc4215540 -1 journal do_read_entry(520192): bad header magic 2024-09-18T15:49:05.965 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:05.959+0000 7fcfc4215540 -1 journal do_read_entry(520192): bad header magic 2024-09-18T15:49:05.973 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:05.967+0000 7fcfc4215540 -1 osd.2 38 log_to_monitors true 2024-09-18T15:49:06.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:07.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:07.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:07.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:49:07.427 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:49:07.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:07.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:49:07.761 INFO:tasks.workunit.client.0.smithi046.stderr:osd.2 up in weight 1 up_from 47 up_thru 47 down_at 39 last_clean_interval [15,38) [v2:127.0.0.1:6818/410175,v1:127.0.0.1:6819/410175] [v2:127.0.0.1:6820/410175,v1:127.0.0.1:6821/410175] exists,up 94b83675-9900-41b4-8639-78375059a3b0 2024-09-18T15:49:07.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:49:07.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:49:07.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:49:07.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:977: list_missing_erasure_coded: create_rbd_pool 2024-09-18T15:49:07.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T15:49:08.190 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' removed 2024-09-18T15:49:08.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T15:49:08.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T15:49:08.777 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' created 2024-09-18T15:49:08.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:49:09.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T15:49:12.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:978: list_missing_erasure_coded: wait_for_clean 2024-09-18T15:49:12.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:49:12.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:49:12.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:49:12.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:49:12.006 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:49:12.007 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:49:12.007 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:49:12.008 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:49:12.008 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:49:12.008 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:49:12.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:49:12.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:49:12.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:49:12.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:49:12.257 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:49:12.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:49:12.576 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:49:12.577 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:49:12.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:49:12.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:49:12.577 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:49:12.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=176093659140 2024-09-18T15:49:12.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 176093659140 2024-09-18T15:49:12.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-176093659140' 2024-09-18T15:49:12.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:49:12.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:49:12.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=188978561027 2024-09-18T15:49:12.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 188978561027 2024-09-18T15:49:12.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-176093659140 1-188978561027' 2024-09-18T15:49:12.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:49:12.914 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:49:13.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=201863462915 2024-09-18T15:49:13.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 201863462915 2024-09-18T15:49:13.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-176093659140 1-188978561027 2-201863462915' 2024-09-18T15:49:13.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:49:13.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-176093659140 2024-09-18T15:49:13.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:49:13.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:49:13.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-176093659140 2024-09-18T15:49:13.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:49:13.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=176093659140 2024-09-18T15:49:13.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 176093659140' 2024-09-18T15:49:13.055 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 176093659140 2024-09-18T15:49:13.056 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:49:13.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 176093659139 -lt 176093659140 2024-09-18T15:49:13.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:49:14.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:49:14.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:49:14.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 176093659140 -lt 176093659140 2024-09-18T15:49:14.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:49:14.689 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-188978561027 2024-09-18T15:49:14.689 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:49:14.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:49:14.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-188978561027 2024-09-18T15:49:14.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:49:14.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=188978561027 2024-09-18T15:49:14.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 188978561027' 2024-09-18T15:49:14.696 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 188978561027 2024-09-18T15:49:14.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:49:15.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 188978561027 -lt 188978561027 2024-09-18T15:49:15.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:49:15.042 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-201863462915 2024-09-18T15:49:15.042 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:49:15.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:49:15.046 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-201863462915 2024-09-18T15:49:15.046 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:49:15.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=201863462915 2024-09-18T15:49:15.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 201863462915' 2024-09-18T15:49:15.049 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 201863462915 2024-09-18T15:49:15.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:49:15.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 201863462915 -lt 201863462915 2024-09-18T15:49:15.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:49:15.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:49:15.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:49:15.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:49:15.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:49:15.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:49:15.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:49:15.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:49:15.793 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:49:15.793 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:49:15.793 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:49:16.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:49:16.144 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:49:16.145 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:49:16.145 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:49:16.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:49:16.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:49:16.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:49:16.518 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-09-18T15:49:16.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-09-18T15:49:16.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=MOBJ0 2024-09-18T15:49:16.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool MOBJ0 2024-09-18T15:49:16.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:49:16.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: local pg=2.0 2024-09-18T15:49:16.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:985: list_missing_erasure_coded: repair 2.0 2024-09-18T15:49:16.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-09-18T15:49:16.912 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-09-18T15:49:16.912 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:49:16.912 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:49:16.912 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:49:16.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:49:17.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T15:48:48.031672+0000 2024-09-18T15:49:17.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-09-18T15:49:17.549 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-09-18T15:49:17.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-09-18T15:48:48.031672+0000 2024-09-18T15:49:17.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=2.0 2024-09-18T15:49:17.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:48:48.031672+0000 2024-09-18T15:49:17.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:49:17.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:49:17.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:49:17.559 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:49:17.559 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:49:17.559 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:49:17.560 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:49:17.560 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:49:17.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:48:48.031672+0000 '>' 2024-09-18T15:48:48.031672+0000 2024-09-18T15:49:17.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:49:18.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:49:18.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:49:18.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:49:18.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:49:18.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:49:18.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:49:18.871 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:49:19.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:48:48.031672+0000 '>' 2024-09-18T15:48:48.031672+0000 2024-09-18T15:49:19.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:49:20.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:49:20.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:49:20.238 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:49:20.238 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:49:20.238 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:49:20.239 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:49:20.239 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:49:20.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:48:48.031672+0000 '>' 2024-09-18T15:48:48.031672+0000 2024-09-18T15:49:20.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:49:21.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:49:21.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:49:21.550 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:49:21.550 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:49:21.550 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:49:21.550 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:49:21.551 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:49:21.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:48:48.031672+0000 '>' 2024-09-18T15:48:48.031672+0000 2024-09-18T15:49:21.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:49:22.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:49:22.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:49:22.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:49:22.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:49:22.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:49:22.871 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:49:22.872 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:49:23.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:49:19.042361+0000 '>' 2024-09-18T15:48:48.031672+0000 2024-09-18T15:49:23.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:49:23.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:987: list_missing_erasure_coded: seq 0 120 2024-09-18T15:49:23.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:987: list_missing_erasure_coded: for i in $(seq 0 120) 2024-09-18T15:49:23.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:988: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-09-18T15:49:23.186 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:989: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-09-18T15:49:23.186 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:989: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-09-18T15:49:23.186 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:989: list_missing_erasure_coded: wc -l 2024-09-18T15:49:23.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:989: list_missing_erasure_coded: matches=2 2024-09-18T15:49:23.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:990: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-09-18T15:49:23.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:990: list_missing_erasure_coded: break 2024-09-18T15:49:23.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T15:49:23.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:49:23.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:49:23.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:49:23.322 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:49:23.323 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:49:23.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:49:23.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:49:23.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:49:23.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:49:23.446 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:49:23.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:49:23.449 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:49:23.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:49:23.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:49:23.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:49:23.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:49:23.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:49:23.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:49:23.456 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:49:23.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:49:23.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:49:23.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:49:23.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:49:23.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:49:23.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:49:23.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:49:23.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:49:23.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:49:23.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T15:49:23.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T15:49:23.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T15:49:23.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T15:49:23.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:49:23.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:49:23.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:49:23.485 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:49:23.485 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:49:23.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:49:23.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:49:23.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:49:23.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:49:23.491 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:49:23.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:49:23.494 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:49:23.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:49:23.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:49:23.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:49:23.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:49:23.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:49:23.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:49:23.501 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:49:23.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:49:23.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:49:23.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:49:23.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:49:23.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:49:23.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:49:23.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:49:23.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:49:23.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:49:23.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T15:49:23.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:49:23.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:49:23.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:49:23.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T15:49:23.517 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:49:23.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:49:23.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:49:23.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T15:49:23.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_list_missing_erasure_coded_overwrites td/osd-scrub-repair 2024-09-18T15:49:23.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: TEST_list_missing_erasure_coded_overwrites: '[' true = true ']' 2024-09-18T15:49:23.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1000: TEST_list_missing_erasure_coded_overwrites: list_missing_erasure_coded td/osd-scrub-repair true 2024-09-18T15:49:23.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:929: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-09-18T15:49:23.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:930: list_missing_erasure_coded: local allow_overwrites=true 2024-09-18T15:49:23.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: local poolname=ecpool 2024-09-18T15:49:23.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:933: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-09-18T15:49:23.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T15:49:23.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:49:23.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:49:23.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:49:23.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T15:49:23.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-18T15:49:23.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:49:23.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:49:23.890 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:49:23.890 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:49:23.890 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:49:23.890 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:49:23.890 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:49:23.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T15:49:24.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:49:24.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:49:24.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:49:24.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:49:24.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:49:24.194 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:49:24.194 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:49:24.194 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:49:24.194 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:49:24.195 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:49:24.195 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:49:24.196 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:49:24.196 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:49:24.197 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:49:24.197 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T15:49:24.298 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:49:24.298 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:49:24.298 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:49:24.298 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:49:24.299 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:49:24.299 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:49:24.299 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:49:24.299 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:49:24.299 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:49:24.299 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:49:24.299 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:49:24.299 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:49:24.299 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:49:24.299 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T15:49:24.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:934: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-09-18T15:49:24.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T15:49:24.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:49:24.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:49:24.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:49:24.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T15:49:24.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:49:25.245 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:49:25.246 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:49:25.246 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:49:25.246 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:49:25.246 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:49:25.246 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:49:25.247 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:49:25.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:49:25.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:49:25.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: seq 0 2 2024-09-18T15:49:25.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-18T15:49:25.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: '[' true = true ']' 2024-09-18T15:49:25.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:937: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-09-18T15:49:25.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:49:25.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:49:25.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:49:25.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:49:25.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:49:25.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:49:25.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:49:25.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:49:25.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:49:25.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:49:25.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:49:25.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:49:25.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:49:25.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:49:25.296 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:49:25.296 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:49:25.296 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:49:25.296 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:49:25.296 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:49:25.296 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:49:25.296 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:49:25.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:49:25.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:49:25.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:49:25.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:49:25.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:49:25.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:49:25.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:49:25.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:49:25.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:49:25.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:49:25.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:49:25.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:49:25.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:49:25.300 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:49:25.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=917a35d1-53f0-4521-a39c-e823158d32d6 2024-09-18T15:49:25.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 917a35d1-53f0-4521-a39c-e823158d32d6' 2024-09-18T15:49:25.302 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 917a35d1-53f0-4521-a39c-e823158d32d6 2024-09-18T15:49:25.302 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:49:25.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCF9upm2oTtEhAAZHjIWUpY8fsuiVhiC/lAVQ== 2024-09-18T15:49:25.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCF9upm2oTtEhAAZHjIWUpY8fsuiVhiC/lAVQ=="}' 2024-09-18T15:49:25.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 917a35d1-53f0-4521-a39c-e823158d32d6 -i td/osd-scrub-repair/0/new.json 2024-09-18T15:49:25.603 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:49:25.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-18T15:49:25.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCF9upm2oTtEhAAZHjIWUpY8fsuiVhiC/lAVQ== --osd-uuid 917a35d1-53f0-4521-a39c-e823158d32d6 2024-09-18T15:49:25.649 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:25.642+0000 7f9d8c184540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:25.696 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:25.690+0000 7f9d8c184540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:25.698 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:25.690+0000 7f9d8c184540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:25.699 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:25.690+0000 7f9d8c184540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:49:25.699 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:25.690+0000 7f9d8c184540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:49:25.699 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:25.690+0000 7f9d8c184540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:49:25.743 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:25.734+0000 7f9d8c184540 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-18T15:49:31.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T15:49:31.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:49:31.743 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T15:49:31.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:49:31.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:49:32.137 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T15:49:32.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:49:32.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:49:32.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:49:32.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:49:32.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:49:32.164 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:32.158+0000 7fdff5d34540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:32.166 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:32.158+0000 7fdff5d34540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:32.168 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:32.162+0000 7fdff5d34540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:32.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:49:32.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:49:32.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:49:32.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:49:32.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:49:32.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:32.419 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:49:32.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:49:32.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:32.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:32.706 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:32.698+0000 7fdff5d34540 -1 Falling back to public interface 2024-09-18T15:49:32.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:33.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:33.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:33.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:49:33.764 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:49:33.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:33.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:34.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:35.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:35.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:35.061 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:49:35.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:49:35.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:35.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:35.355 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:35.346+0000 7fdff5d34540 -1 osd.0 0 log_to_monitors true 2024-09-18T15:49:35.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:36.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:36.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:36.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:49:36.375 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:49:36.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:36.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:36.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:37.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:37.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:37.689 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:49:37.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:49:37.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:37.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:38.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:39.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:39.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:39.157 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:49:39.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:49:39.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:39.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:39.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:40.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:40.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:40.474 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:49:40.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:49:40.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:40.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:40.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:41.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:41.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:41.794 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:49:41.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:49:41.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:41.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:42.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:43.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:43.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:43.161 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:49:43.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:49:43.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:43.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:43.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:44.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:44.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:44.482 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:49:44.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:49:44.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:44.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:44.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:45.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:45.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:45.800 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:49:45.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:49:45.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:45.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:46.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:47.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:47.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:47.157 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:49:47.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:49:47.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:47.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:47.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:48.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:48.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:48.730 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:49:48.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:49:48.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:48.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:49.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:50.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:50.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:50.217 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:49:50.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:49:50.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:50.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:50.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:51.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:51.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:51.667 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:49:51.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:49:51.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:51.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:52.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:53.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:53.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:53.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:49:53.324 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:49:53.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:53.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:53.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:54.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:54.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:54.677 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:49:54.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:49:54.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:54.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:55.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:56.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:56.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:56.115 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:49:56.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:49:56.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:56.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:56.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:57.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:57.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:57.478 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:49:57.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:49:57.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:57.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:57.861 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/412150,v1:127.0.0.1:6803/412150] [v2:127.0.0.1:6804/412150,v1:127.0.0.1:6805/412150] exists,up 917a35d1-53f0-4521-a39c-e823158d32d6 2024-09-18T15:49:57.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:49:57.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:49:57.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:49:57.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-18T15:49:57.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: '[' true = true ']' 2024-09-18T15:49:57.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:937: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-09-18T15:49:57.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:49:57.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:49:57.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:49:57.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:49:57.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:49:57.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:49:57.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:49:57.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:49:57.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:49:57.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:49:57.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:49:57.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:49:57.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:49:57.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:49:57.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:49:57.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:49:57.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:49:57.867 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:49:57.867 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:49:57.867 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:49:57.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:49:57.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:49:57.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:49:57.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:49:57.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:49:57.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:49:57.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:49:57.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:49:57.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:49:57.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:49:57.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:49:57.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:49:57.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:49:57.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:49:57.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:49:57.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0048ac94-3a72-4300-910b-c90e1129bab1 2024-09-18T15:49:57.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0048ac94-3a72-4300-910b-c90e1129bab1' 2024-09-18T15:49:57.871 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 0048ac94-3a72-4300-910b-c90e1129bab1 2024-09-18T15:49:57.872 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:49:57.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCl9upm+oEoNRAA4kUA4VyGHeIlk3ly5DpRNA== 2024-09-18T15:49:57.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCl9upm+oEoNRAA4kUA4VyGHeIlk3ly5DpRNA=="}' 2024-09-18T15:49:57.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0048ac94-3a72-4300-910b-c90e1129bab1 -i td/osd-scrub-repair/1/new.json 2024-09-18T15:49:58.332 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:49:58.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-18T15:49:58.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCl9upm+oEoNRAA4kUA4VyGHeIlk3ly5DpRNA== --osd-uuid 0048ac94-3a72-4300-910b-c90e1129bab1 2024-09-18T15:49:58.383 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:58.378+0000 7f1bc379e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:58.456 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:58.442+0000 7f1bc379e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:58.458 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:58.450+0000 7f1bc379e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:58.459 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:58.450+0000 7f1bc379e540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:49:58.459 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:58.450+0000 7f1bc379e540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:49:58.459 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:58.450+0000 7f1bc379e540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:49:58.607 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:49:58.602+0000 7f1bc379e540 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-18T15:50:04.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-18T15:50:04.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:50:04.926 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T15:50:04.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:50:04.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:50:05.326 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T15:50:05.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:50:05.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:50:05.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:50:05.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:50:05.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:50:05.358 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:50:05.350+0000 7f544e41d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:50:05.377 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:50:05.370+0000 7f544e41d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:50:05.378 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:50:05.370+0000 7f544e41d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:50:05.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:50:05.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:50:05.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:50:05.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:50:05.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:50:05.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:05.658 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:50:05.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:50:05.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:05.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:50:05.926 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:50:05.918+0000 7f544e41d540 -1 Falling back to public interface 2024-09-18T15:50:06.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:07.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:07.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:07.011 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:50:07.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:50:07.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:07.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:50:07.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:08.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:08.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:08.330 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:50:08.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:50:08.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:08.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:50:08.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:08.722 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:50:08.698+0000 7f544e41d540 -1 osd.1 0 log_to_monitors true 2024-09-18T15:50:09.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:09.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:09.649 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:50:09.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:50:09.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:09.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:50:09.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:10.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:10.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:10.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:50:10.999 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:50:10.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:10.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:50:11.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:12.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:12.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:12.325 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:50:12.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:50:12.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:12.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:50:12.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:13.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:13.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:13.768 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:50:13.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:50:13.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:13.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:50:14.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:15.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:15.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:15.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:50:15.091 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:50:15.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:15.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:50:15.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:16.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:16.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:16.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:50:16.444 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:50:16.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:16.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:50:16.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:17.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:17.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:17.817 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:50:17.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:50:17.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:17.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:50:18.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:19.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:19.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:19.136 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:50:19.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:50:19.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:19.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:50:19.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:20.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:20.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:20.447 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:50:20.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:50:20.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:20.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:50:21.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:22.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:22.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:22.114 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:50:22.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:50:22.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:22.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:50:22.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:23.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:23.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:23.674 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:50:23.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:50:23.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:23.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:50:24.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:25.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:25.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:25.029 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:50:25.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:50:25.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:25.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:50:25.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:26.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:26.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:26.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:50:26.486 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:50:26.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:26.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:50:27.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:28.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:28.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:28.050 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:50:28.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:50:28.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:28.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:50:28.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:29.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:29.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:29.596 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:50:29.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:50:29.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:29.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:50:30.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:31.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:31.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:31.062 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:50:31.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:50:31.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:31.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:50:31.391 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/413930,v1:127.0.0.1:6811/413930] [v2:127.0.0.1:6812/413930,v1:127.0.0.1:6813/413930] exists,up 0048ac94-3a72-4300-910b-c90e1129bab1 2024-09-18T15:50:31.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:50:31.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:50:31.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:50:31.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-18T15:50:31.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: '[' true = true ']' 2024-09-18T15:50:31.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:937: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-09-18T15:50:31.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:50:31.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:50:31.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T15:50:31.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:50:31.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:50:31.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:50:31.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:50:31.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:50:31.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:50:31.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T15:50:31.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T15:50:31.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:50:31.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:50:31.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:50:31.395 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:50:31.395 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:50:31.395 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:50:31.396 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:50:31.396 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:50:31.396 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:50:31.396 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:50:31.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:50:31.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:50:31.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:50:31.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:50:31.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:50:31.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:50:31.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:50:31.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:50:31.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:50:31.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:50:31.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:50:31.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:50:31.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-18T15:50:31.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:50:31.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fc55c1ad-22fd-4a36-b668-2445937a7b56 2024-09-18T15:50:31.399 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 fc55c1ad-22fd-4a36-b668-2445937a7b56 2024-09-18T15:50:31.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 fc55c1ad-22fd-4a36-b668-2445937a7b56' 2024-09-18T15:50:31.400 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:50:31.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDH9upmijykGBAAcO/5TQmckVUVeq/AA/I/6A== 2024-09-18T15:50:31.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDH9upmijykGBAAcO/5TQmckVUVeq/AA/I/6A=="}' 2024-09-18T15:50:31.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fc55c1ad-22fd-4a36-b668-2445937a7b56 -i td/osd-scrub-repair/2/new.json 2024-09-18T15:50:31.857 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:50:31.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-18T15:50:31.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDH9upmijykGBAAcO/5TQmckVUVeq/AA/I/6A== --osd-uuid fc55c1ad-22fd-4a36-b668-2445937a7b56 2024-09-18T15:50:31.905 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:50:31.898+0000 7fedacead540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:50:31.914 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:50:31.906+0000 7fedacead540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:50:31.916 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:50:31.910+0000 7fedacead540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:50:31.917 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:50:31.910+0000 7fedacead540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T15:50:31.918 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:50:31.910+0000 7fedacead540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T15:50:31.918 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:50:31.910+0000 7fedacead540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T15:50:31.997 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:50:31.990+0000 7fedacead540 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-18T15:50:38.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-18T15:50:38.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:50:38.496 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T15:50:38.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T15:50:38.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:50:38.880 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T15:50:38.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T15:50:38.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:50:38.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:50:38.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:50:38.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:50:38.910 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:50:38.902+0000 7f163cfda540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:50:38.912 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:50:38.906+0000 7f163cfda540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:50:38.915 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:50:38.906+0000 7f163cfda540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:50:39.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T15:50:39.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:50:39.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:50:39.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:50:39.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:50:39.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:39.201 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:50:39.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:50:39.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:39.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:39.450 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:50:39.442+0000 7f163cfda540 -1 Falling back to public interface 2024-09-18T15:50:39.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:40.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:40.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:40.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:50:40.513 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:50:40.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:40.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:40.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:41.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:41.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:41.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:50:41.826 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:50:41.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:41.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:42.094 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:50:42.086+0000 7f163cfda540 -1 osd.2 0 log_to_monitors true 2024-09-18T15:50:42.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:43.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:43.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:43.140 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:50:43.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:50:43.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:43.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:43.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:44.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:44.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:44.469 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:50:44.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:50:44.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:44.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:45.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:46.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:46.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:46.040 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:50:46.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:50:46.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:46.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:46.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:47.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:47.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:47.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:50:47.357 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:50:47.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:47.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:47.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:48.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:48.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:48.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:50:48.674 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:50:48.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:48.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:49.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:50.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:50.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:50.188 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:50:50.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:50:50.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:50.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:50.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:51.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:51.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:51.510 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:50:51.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:50:51.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:51.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:51.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:52.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:52.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:52.958 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:50:52.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:50:52.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:52.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:53.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:54.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:54.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:54.266 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:50:54.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:50:54.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:54.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:54.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:55.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:55.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:55.581 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:50:55.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:50:55.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:55.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:56.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:57.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:57.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:57.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:50:57.258 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:50:57.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:57.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:57.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:58.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:58.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:58.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:50:58.613 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:50:58.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:58.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:59.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:51:00.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:51:00.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:51:00.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:51:00.044 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:51:00.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:51:00.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:51:00.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:51:01.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:51:01.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:51:01.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:51:01.545 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:51:01.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:51:01.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:51:02.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:51:03.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:51:03.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:51:03.264 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:51:03.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:51:03.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:51:03.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:51:03.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:51:04.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:51:04.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:51:04.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:51:04.973 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:51:04.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:51:04.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:51:05.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:51:06.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:51:06.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:51:06.417 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:51:06.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:51:06.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:51:06.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:51:06.736 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/415709,v1:127.0.0.1:6819/415709] [v2:127.0.0.1:6820/415709,v1:127.0.0.1:6821/415709] exists,up fc55c1ad-22fd-4a36-b668-2445937a7b56 2024-09-18T15:51:06.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:51:06.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:51:06.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:51:06.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:942: list_missing_erasure_coded: create_rbd_pool 2024-09-18T15:51:06.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T15:51:07.097 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' does not exist 2024-09-18T15:51:07.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T15:51:07.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T15:51:07.731 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' created 2024-09-18T15:51:07.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:51:08.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T15:51:10.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: wait_for_clean 2024-09-18T15:51:10.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:51:10.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:51:10.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:51:10.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:51:10.874 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:51:10.875 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:51:10.875 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:51:10.876 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:51:10.876 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:51:10.876 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:51:11.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:51:11.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:51:11.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:51:11.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:51:11.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:51:11.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:51:11.429 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:51:11.429 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:51:11.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:51:11.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:51:11.430 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:51:11.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836500 2024-09-18T15:51:11.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836500 2024-09-18T15:51:11.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500' 2024-09-18T15:51:11.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:51:11.589 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:51:11.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672975 2024-09-18T15:51:11.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672975 2024-09-18T15:51:11.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672975' 2024-09-18T15:51:11.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:51:11.723 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:51:11.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509446 2024-09-18T15:51:11.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509446 2024-09-18T15:51:11.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672975 2-64424509446' 2024-09-18T15:51:11.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:51:11.858 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836500 2024-09-18T15:51:11.858 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:51:11.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:51:11.862 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836500 2024-09-18T15:51:11.862 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:51:11.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836500 2024-09-18T15:51:11.865 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836500 2024-09-18T15:51:11.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-18T15:51:11.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:51:12.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836499 -lt 21474836500 2024-09-18T15:51:12.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:51:13.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:51:13.165 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:51:13.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836500 -lt 21474836500 2024-09-18T15:51:13.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:51:13.487 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672975 2024-09-18T15:51:13.488 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:51:13.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:51:13.492 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672975 2024-09-18T15:51:13.492 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:51:13.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672975 2024-09-18T15:51:13.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-18T15:51:13.494 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672975 2024-09-18T15:51:13.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:51:13.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672976 -lt 42949672975 2024-09-18T15:51:13.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:51:13.844 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509446 2024-09-18T15:51:13.844 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:51:13.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:51:13.848 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509446 2024-09-18T15:51:13.849 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:51:13.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509446 2024-09-18T15:51:13.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-09-18T15:51:13.850 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509446 2024-09-18T15:51:13.851 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:51:14.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509446 -lt 64424509446 2024-09-18T15:51:14.189 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:51:14.190 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:51:14.190 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:51:14.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T15:51:14.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:51:14.608 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:51:14.608 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:51:14.608 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:51:14.609 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:51:14.609 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:51:14.609 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:51:14.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T15:51:14.977 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:51:14.978 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:51:14.978 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:51:15.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T15:51:15.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:51:15.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:51:15.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:945: list_missing_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-09-18T15:51:15.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: create_ec_pool: local pool_name=ecpool 2024-09-18T15:51:15.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: create_ec_pool: shift 2024-09-18T15:51:15.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: create_ec_pool: local allow_overwrites=true 2024-09-18T15:51:15.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: create_ec_pool: shift 2024-09-18T15:51:15.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-18T15:51:15.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-18T15:51:15.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-18T15:51:16.502 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'ecpool' created 2024-09-18T15:51:16.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:51:17.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: create_ec_pool: '[' true = true ']' 2024-09-18T15:51:17.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-18T15:51:17.957 INFO:tasks.workunit.client.0.smithi046.stderr:set pool 2 allow_ec_overwrites to true 2024-09-18T15:51:17.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: create_ec_pool: wait_for_clean 2024-09-18T15:51:17.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:51:17.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:51:17.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:51:17.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:51:17.974 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:51:17.974 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:51:17.974 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:51:17.975 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:51:17.975 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:51:17.975 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:51:18.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:51:18.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:51:18.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:51:18.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:51:18.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:51:18.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:51:18.524 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:51:18.524 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:51:18.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:51:18.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:51:18.525 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:51:18.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836502 2024-09-18T15:51:18.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836502 2024-09-18T15:51:18.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836502' 2024-09-18T15:51:18.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:51:18.725 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:51:18.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672978 2024-09-18T15:51:18.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672978 2024-09-18T15:51:18.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836502 1-42949672978' 2024-09-18T15:51:18.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:51:18.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:51:19.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509448 2024-09-18T15:51:19.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509448 2024-09-18T15:51:19.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836502 1-42949672978 2-64424509448' 2024-09-18T15:51:19.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:51:19.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836502 2024-09-18T15:51:19.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:51:19.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:51:19.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836502 2024-09-18T15:51:19.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:51:19.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836502 2024-09-18T15:51:19.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-18T15:51:19.062 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836502 2024-09-18T15:51:19.063 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:51:19.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836501 -lt 21474836502 2024-09-18T15:51:19.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:51:20.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:51:20.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:51:20.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836501 -lt 21474836502 2024-09-18T15:51:20.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:51:21.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-18T15:51:21.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:51:22.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836503 -lt 21474836502 2024-09-18T15:51:22.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:51:22.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672978 2024-09-18T15:51:22.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:51:22.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:51:22.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672978 2024-09-18T15:51:22.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:51:22.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672978 2024-09-18T15:51:22.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-09-18T15:51:22.032 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672978 2024-09-18T15:51:22.033 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:51:22.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672978 -lt 42949672978 2024-09-18T15:51:22.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:51:22.374 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509448 2024-09-18T15:51:22.374 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:51:22.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:51:22.378 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509448 2024-09-18T15:51:22.378 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:51:22.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509448 2024-09-18T15:51:22.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-18T15:51:22.381 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509448 2024-09-18T15:51:22.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:51:22.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509448 -lt 64424509448 2024-09-18T15:51:22.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:51:22.758 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:51:22.758 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:51:23.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:51:23.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:51:23.156 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:51:23.156 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:51:23.156 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:51:23.156 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:51:23.157 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:51:23.157 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:51:23.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:51:23.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:51:23.513 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:51:23.513 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:51:23.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:51:23.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:51:23.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:51:23.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: create_ec_pool: return 0 2024-09-18T15:51:23.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:948: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-09-18T15:51:23.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:51:23.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2024-09-18T15:51:23.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=MOBJ0 2024-09-18T15:51:23.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:51:23.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:51:23.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:51:24.479 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:51:24.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:51:25.075 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:51:25.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:51:25.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:51:25.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:51:25.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:949: list_missing_erasure_coded: osds0=($(get_osds $poolname MOBJ0)) 2024-09-18T15:51:25.333 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:949: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-09-18T15:51:25.333 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-09-18T15:51:25.333 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ0 2024-09-18T15:51:25.334 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ0 2024-09-18T15:51:25.334 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:51:25.668 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-18T15:51:25.668 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:51:25.668 INFO:tasks.workunit.client.0.smithi046.stderr:0' 2024-09-18T15:51:25.668 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-09-18T15:51:25.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:949: list_missing_erasure_coded: local -a osds0 2024-09-18T15:51:25.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:952: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-09-18T15:51:25.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:51:25.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2024-09-18T15:51:25.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=MOBJ1 2024-09-18T15:51:25.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T15:51:25.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T15:51:25.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T15:51:26.152 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T15:51:26.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T15:51:26.700 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:51:26.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:51:26.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:51:26.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:51:26.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: osds1=($(get_osds $poolname MOBJ1)) 2024-09-18T15:51:26.919 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-09-18T15:51:26.919 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-09-18T15:51:26.919 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ1 2024-09-18T15:51:26.921 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ1 2024-09-18T15:51:26.921 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:51:27.282 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-18T15:51:27.282 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:51:27.282 INFO:tasks.workunit.client.0.smithi046.stderr:0' 2024-09-18T15:51:27.282 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-09-18T15:51:27.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: local -a osds1 2024-09-18T15:51:27.283 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:956: list_missing_erasure_coded: seq 0 2 2024-09-18T15:51:27.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:956: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-18T15:51:27.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:51:27.286 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:51:27.286 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:51:27.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:51:27.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:51:27.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:51:27.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:51:27.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:956: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-18T15:51:27.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:51:27.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:51:27.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:51:27.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:51:27.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:51:27.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:51:28.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:51:28.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:956: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-18T15:51:28.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-18T15:51:28.924 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:51:28.924 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:51:28.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:51:28.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:51:28.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:51:29.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:51:29.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: id=2 2024-09-18T15:51:29.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-09-18T15:51:30.705 INFO:tasks.workunit.client.0.smithi046.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-09-18T15:51:31.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:963: list_missing_erasure_coded: id=1 2024-09-18T15:51:31.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-09-18T15:51:32.594 INFO:tasks.workunit.client.0.smithi046.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-09-18T15:51:33.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:967: list_missing_erasure_coded: id=1 2024-09-18T15:51:33.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-09-18T15:51:34.077 INFO:tasks.workunit.client.0.smithi046.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-09-18T15:51:34.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: id=0 2024-09-18T15:51:34.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-09-18T15:51:36.124 INFO:tasks.workunit.client.0.smithi046.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-09-18T15:51:36.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:974: list_missing_erasure_coded: seq 0 2 2024-09-18T15:51:36.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:974: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-18T15:51:36.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:975: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-09-18T15:51:36.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:51:36.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:51:36.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:51:36.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:51:36.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:51:36.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:51:36.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:51:36.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:51:36.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:51:36.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:51:36.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:51:36.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:51:36.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:51:36.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:51:36.679 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:51:36.679 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:51:36.679 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:51:36.679 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:51:36.679 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:51:36.679 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:51:36.679 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:51:36.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:51:36.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:51:36.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:51:36.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:51:36.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:51:36.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:51:36.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:51:36.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:51:36.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:51:36.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:51:36.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:51:36.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:51:36.682 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:51:36.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:51:36.683 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:51:36.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:51:36.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:51:36.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:51:36.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:51:36.714 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:51:36.705+0000 7fa6fac18540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:51:36.731 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:51:36.721+0000 7fa6fac18540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:51:36.733 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:51:36.725+0000 7fa6fac18540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:51:36.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:51:36.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:51:36.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:51:36.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:51:36.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:51:36.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:51:36.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:51:36.994 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:51:36.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:51:36.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:51:37.269 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:51:37.261+0000 7fa6fac18540 -1 Falling back to public interface 2024-09-18T15:51:37.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:51:38.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:51:38.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:51:38.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:51:38.304 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:51:38.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:51:38.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:51:38.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:51:39.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:51:39.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:51:39.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:51:39.623 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:51:39.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:51:39.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:51:39.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:51:40.096 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:51:40.089+0000 7fa6fac18540 -1 osd.0 35 log_to_monitors true 2024-09-18T15:51:40.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:51:40.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:51:40.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:51:40.941 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:51:40.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:51:40.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:51:41.302 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 42 up_thru 42 down_at 36 last_clean_interval [5,35) [v2:127.0.0.1:6802/419365,v1:127.0.0.1:6803/419365] [v2:127.0.0.1:6804/419365,v1:127.0.0.1:6805/419365] exists,up 917a35d1-53f0-4521-a39c-e823158d32d6 2024-09-18T15:51:41.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:51:41.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:51:41.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:51:41.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:974: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-18T15:51:41.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:975: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-09-18T15:51:41.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:51:41.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:51:41.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:51:41.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:51:41.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:51:41.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:51:41.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:51:41.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:51:41.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:51:41.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:51:41.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:51:41.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:51:41.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:51:41.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:51:41.306 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:51:41.306 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:51:41.306 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:51:41.306 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:51:41.306 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:51:41.306 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:51:41.306 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:51:41.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:51:41.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:51:41.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:51:41.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:51:41.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:51:41.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:51:41.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:51:41.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:51:41.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:51:41.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:51:41.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:51:41.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:51:41.308 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T15:51:41.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:51:41.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:51:41.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:51:41.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:51:41.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:51:41.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:51:41.338 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:51:41.329+0000 7f80814ae540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:51:41.366 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:51:41.357+0000 7f80814ae540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:51:41.368 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:51:41.361+0000 7f80814ae540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:51:41.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:51:41.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:51:41.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:51:41.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:51:41.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:51:41.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:51:41.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:51:41.643 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:51:41.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:51:41.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:51:41.909 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:51:41.901+0000 7f80814ae540 -1 Falling back to public interface 2024-09-18T15:51:41.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:51:42.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:51:42.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:51:42.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:51:42.992 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:51:42.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:51:42.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:51:43.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:51:44.221 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:51:44.213+0000 7f80814ae540 -1 osd.1 36 log_to_monitors true 2024-09-18T15:51:44.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:51:44.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:51:44.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:51:44.304 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:51:44.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:51:44.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:51:44.623 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 45 up_thru 36 down_at 37 last_clean_interval [10,36) [v2:127.0.0.1:6810/419943,v1:127.0.0.1:6811/419943] [v2:127.0.0.1:6812/419943,v1:127.0.0.1:6813/419943] exists,up 0048ac94-3a72-4300-910b-c90e1129bab1 2024-09-18T15:51:44.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:51:44.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:51:44.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:51:44.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:974: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-18T15:51:44.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:975: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-09-18T15:51:44.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:51:44.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:51:44.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T15:51:44.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:51:44.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:51:44.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:51:44.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:51:44.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:51:44.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:51:44.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T15:51:44.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T15:51:44.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:51:44.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:51:44.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:51:44.626 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:51:44.626 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:51:44.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:51:44.627 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:51:44.627 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:51:44.627 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:51:44.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:51:44.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:51:44.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:51:44.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:51:44.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:51:44.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:51:44.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:51:44.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:51:44.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:51:44.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:51:44.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:51:44.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-18T15:51:44.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T15:51:44.628 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.2 2024-09-18T15:51:44.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:51:44.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-18T15:51:44.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T15:51:44.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:51:44.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:51:44.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:51:44.658 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:51:44.649+0000 7fc295828540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:51:44.705 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:51:44.697+0000 7fc295828540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:51:44.707 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:51:44.697+0000 7fc295828540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:51:45.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T15:51:45.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:51:45.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:51:45.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:51:45.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:51:45.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:51:45.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:51:45.107 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:51:45.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:51:45.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:51:45.270 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:51:45.261+0000 7fc295828540 -1 Falling back to public interface 2024-09-18T15:51:45.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:51:46.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:51:46.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:51:46.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:51:46.530 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:51:46.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:51:46.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:51:46.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:51:47.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:51:47.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:51:47.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:51:47.840 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:51:47.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:51:47.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:51:48.114 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:51:48.105+0000 7fc295828540 -1 osd.2 39 log_to_monitors true 2024-09-18T15:51:48.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:51:49.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:51:49.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:51:49.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:51:49.157 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:51:49.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:51:49.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:51:49.549 INFO:tasks.workunit.client.0.smithi046.stderr:osd.2 up in weight 1 up_from 49 up_thru 49 down_at 40 last_clean_interval [15,39) [v2:127.0.0.1:6818/420497,v1:127.0.0.1:6819/420497] [v2:127.0.0.1:6820/420497,v1:127.0.0.1:6821/420497] exists,up fc55c1ad-22fd-4a36-b668-2445937a7b56 2024-09-18T15:51:49.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:51:49.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:51:49.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:51:49.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:977: list_missing_erasure_coded: create_rbd_pool 2024-09-18T15:51:49.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T15:51:50.060 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' removed 2024-09-18T15:51:50.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T15:51:50.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T15:51:50.585 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' created 2024-09-18T15:51:50.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:51:51.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T15:51:53.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:978: list_missing_erasure_coded: wait_for_clean 2024-09-18T15:51:53.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:51:53.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:51:53.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:51:53.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:51:53.384 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:51:53.386 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:51:53.386 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:51:53.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:51:53.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:51:53.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:51:53.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:51:53.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:51:53.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:51:53.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:51:53.620 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:51:53.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:51:53.943 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:51:53.943 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:51:53.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:51:53.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:51:53.944 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:51:54.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=180388626436 2024-09-18T15:51:54.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 180388626436 2024-09-18T15:51:54.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-180388626436' 2024-09-18T15:51:54.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:51:54.132 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:51:54.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528323 2024-09-18T15:51:54.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528323 2024-09-18T15:51:54.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-180388626436 1-193273528323' 2024-09-18T15:51:54.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:51:54.276 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:51:54.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=210453397507 2024-09-18T15:51:54.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 210453397507 2024-09-18T15:51:54.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-180388626436 1-193273528323 2-210453397507' 2024-09-18T15:51:54.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:51:54.417 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-180388626436 2024-09-18T15:51:54.417 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:51:54.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:51:54.421 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-180388626436 2024-09-18T15:51:54.421 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:51:54.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=180388626436 2024-09-18T15:51:54.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 180388626436' 2024-09-18T15:51:54.424 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 180388626436 2024-09-18T15:51:54.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:51:54.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 180388626435 -lt 180388626436 2024-09-18T15:51:54.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:51:55.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:51:55.748 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:51:56.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 180388626436 -lt 180388626436 2024-09-18T15:51:56.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:51:56.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-193273528323 2024-09-18T15:51:56.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:51:56.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:51:56.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-193273528323 2024-09-18T15:51:56.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:51:56.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528323 2024-09-18T15:51:56.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 193273528323' 2024-09-18T15:51:56.072 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 193273528323 2024-09-18T15:51:56.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:51:56.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528324 -lt 193273528323 2024-09-18T15:51:56.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:51:56.412 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:51:56.412 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-210453397507 2024-09-18T15:51:56.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:51:56.415 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-210453397507 2024-09-18T15:51:56.415 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:51:56.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=210453397507 2024-09-18T15:51:56.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 210453397507' 2024-09-18T15:51:56.417 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 210453397507 2024-09-18T15:51:56.418 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:51:56.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 210453397507 -lt 210453397507 2024-09-18T15:51:56.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:51:56.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:51:56.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:51:57.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:51:57.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:51:57.156 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:51:57.156 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:51:57.157 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:51:57.157 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:51:57.157 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:51:57.157 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:51:57.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:51:57.523 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:51:57.524 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:51:57.524 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:51:57.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:51:57.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:51:57.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:51:57.879 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-09-18T15:51:57.879 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-09-18T15:51:57.879 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=MOBJ0 2024-09-18T15:51:57.880 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:51:57.880 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool MOBJ0 2024-09-18T15:51:58.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: local pg=2.0 2024-09-18T15:51:58.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:985: list_missing_erasure_coded: repair 2.0 2024-09-18T15:51:58.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=2.0 2024-09-18T15:51:58.236 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 2.0 2024-09-18T15:51:58.236 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:51:58.236 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:51:58.236 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:51:58.236 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:51:58.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T15:51:16.410596+0000 2024-09-18T15:51:58.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 2.0 2024-09-18T15:51:58.861 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-09-18T15:51:58.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 2.0 2024-09-18T15:51:16.410596+0000 2024-09-18T15:51:58.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=2.0 2024-09-18T15:51:58.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:51:16.410596+0000 2024-09-18T15:51:58.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:51:58.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:51:58.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:51:58.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:51:58.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:51:58.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:51:58.871 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:51:58.871 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:51:59.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:51:16.410596+0000 '>' 2024-09-18T15:51:16.410596+0000 2024-09-18T15:51:59.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:52:00.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:52:00.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:52:00.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:52:00.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:52:00.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:52:00.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:52:00.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:52:00.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:51:16.410596+0000 '>' 2024-09-18T15:51:16.410596+0000 2024-09-18T15:52:00.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:52:01.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:52:01.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:52:01.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:52:01.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:52:01.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:52:01.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:52:01.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:52:01.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:51:16.410596+0000 '>' 2024-09-18T15:51:16.410596+0000 2024-09-18T15:52:01.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:52:02.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:52:02.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:52:02.794 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:52:02.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:52:02.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:52:02.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:52:02.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:52:03.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:51:16.410596+0000 '>' 2024-09-18T15:51:16.410596+0000 2024-09-18T15:52:03.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:52:04.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:52:04.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:52:04.114 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:52:04.114 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:52:04.114 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:52:04.114 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:52:04.115 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:52:04.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:51:16.410596+0000 '>' 2024-09-18T15:51:16.410596+0000 2024-09-18T15:52:04.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:52:05.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:52:05.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:52:05.436 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:52:05.436 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:52:05.436 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:52:05.437 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:52:05.437 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:52:05.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:52:00.162109+0000 '>' 2024-09-18T15:51:16.410596+0000 2024-09-18T15:52:05.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:52:05.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:987: list_missing_erasure_coded: seq 0 120 2024-09-18T15:52:05.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:987: list_missing_erasure_coded: for i in $(seq 0 120) 2024-09-18T15:52:05.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:988: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-09-18T15:52:05.745 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:989: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-09-18T15:52:05.745 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:989: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-09-18T15:52:05.745 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:989: list_missing_erasure_coded: wc -l 2024-09-18T15:52:05.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:989: list_missing_erasure_coded: matches=2 2024-09-18T15:52:05.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:990: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-09-18T15:52:05.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:990: list_missing_erasure_coded: break 2024-09-18T15:52:05.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T15:52:05.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:52:05.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:52:05.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:52:05.887 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:52:05.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:52:05.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:52:05.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:52:05.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:52:06.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:52:06.009 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:52:06.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:52:06.012 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:52:06.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:52:06.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:52:06.016 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:52:06.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:52:06.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:52:06.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:52:06.020 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:52:06.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:52:06.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:52:06.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:52:06.047 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:52:06.047 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:52:06.048 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:52:06.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:52:06.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:52:06.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:52:06.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T15:52:06.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T15:52:06.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T15:52:06.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T15:52:06.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:52:06.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:52:06.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:52:06.051 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:52:06.051 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:52:06.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:52:06.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:52:06.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:52:06.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:52:06.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:52:06.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:52:06.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:52:06.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:52:06.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:52:06.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:52:06.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:52:06.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:52:06.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:52:06.069 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:52:06.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:52:06.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:52:06.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:52:06.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:52:06.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:52:06.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:52:06.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:52:06.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:52:06.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:52:06.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T15:52:06.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:52:06.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:52:06.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:52:06.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T15:52:06.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:52:06.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:52:06.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:52:06.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T15:52:06.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_periodic_scrub_replicated td/osd-scrub-repair 2024-09-18T15:52:06.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5780: TEST_periodic_scrub_replicated: local dir=td/osd-scrub-repair 2024-09-18T15:52:06.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5781: TEST_periodic_scrub_replicated: local poolname=psr_pool 2024-09-18T15:52:06.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5782: TEST_periodic_scrub_replicated: local objname=POBJ 2024-09-18T15:52:06.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5784: TEST_periodic_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-09-18T15:52:06.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T15:52:06.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:52:06.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:52:06.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:52:06.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T15:52:06.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-09-18T15:52:06.455 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:52:06.455 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:52:06.456 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:52:06.456 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:52:06.456 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:52:06.456 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:52:06.457 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:52:06.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-18T15:52:06.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:52:06.755 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:52:06.755 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:52:06.755 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:52:06.755 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:52:06.756 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:52:06.757 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:52:06.757 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:52:06.757 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:52:06.758 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:52:06.758 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:52:06.758 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:52:06.759 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:52:06.759 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:52:06.760 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T15:52:06.851 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:52:06.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:52:06.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:52:06.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:52:06.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:52:06.853 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:52:06.853 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:52:06.853 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:52:06.854 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:52:06.854 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:52:06.854 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:52:06.854 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:52:06.855 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:52:06.855 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T15:52:06.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5785: TEST_periodic_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-09-18T15:52:06.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T15:52:06.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:52:06.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:52:06.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:52:06.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T15:52:06.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:52:07.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:52:07.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:52:07.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:52:07.806 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:52:07.806 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:52:07.806 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:52:07.806 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:52:07.808 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:52:07.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:52:07.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5786: TEST_periodic_scrub_replicated: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2024-09-18T15:52:07.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5787: TEST_periodic_scrub_replicated: ceph_osd_args+=--osd_scrub_backoff_ratio=0 2024-09-18T15:52:07.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5788: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-09-18T15:52:07.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:52:07.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:52:07.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:52:07.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:52:07.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:52:07.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:52:07.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:52:07.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:52:07.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:52:07.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:52:07.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:52:07.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:52:07.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:52:07.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:52:07.847 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:52:07.848 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:52:07.848 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:52:07.848 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:52:07.848 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:52:07.848 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:52:07.848 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:52:07.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:52:07.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:52:07.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:52:07.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:52:07.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:52:07.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:52:07.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:52:07.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:52:07.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:52:07.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:52:07.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:52:07.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-09-18T15:52:07.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:52:07.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:52:07.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6b2dfba1-d6b2-4e6a-abac-aeb79442f721 2024-09-18T15:52:07.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6b2dfba1-d6b2-4e6a-abac-aeb79442f721' 2024-09-18T15:52:07.853 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 6b2dfba1-d6b2-4e6a-abac-aeb79442f721 2024-09-18T15:52:07.855 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:52:07.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAn9+pmq9XdMxAAcO+PFdnXusgP7F5WTrwAlQ== 2024-09-18T15:52:07.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAn9+pmq9XdMxAAcO+PFdnXusgP7F5WTrwAlQ=="}' 2024-09-18T15:52:07.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6b2dfba1-d6b2-4e6a-abac-aeb79442f721 -i td/osd-scrub-repair/0/new.json 2024-09-18T15:52:08.143 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:52:08.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-18T15:52:08.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQAn9+pmq9XdMxAAcO+PFdnXusgP7F5WTrwAlQ== --osd-uuid 6b2dfba1-d6b2-4e6a-abac-aeb79442f721 2024-09-18T15:52:08.187 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:52:08.177+0000 7f85e77ac540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:08.192 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:52:08.185+0000 7f85e77ac540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:08.193 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:52:08.185+0000 7f85e77ac540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:08.194 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:52:08.185+0000 7f85e77ac540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:52:08.194 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:52:08.185+0000 7f85e77ac540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:52:08.194 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:52:08.185+0000 7f85e77ac540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:52:08.270 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:52:08.261+0000 7f85e77ac540 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-18T15:52:14.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T15:52:14.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:52:14.303 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T15:52:14.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:52:14.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:52:14.689 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T15:52:14.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:52:14.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-09-18T15:52:14.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:52:14.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:52:14.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:52:14.721 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:52:14.713+0000 7fb97c308540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:14.723 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:52:14.717+0000 7fb97c308540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:14.725 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:52:14.717+0000 7fb97c308540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:15.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:52:15.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:52:15.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:52:15.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:52:15.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:52:15.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:15.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:52:15.035 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:52:15.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:15.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:52:15.290 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:52:15.281+0000 7fb97c308540 -1 Falling back to public interface 2024-09-18T15:52:15.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:16.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:16.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:16.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:52:16.454 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:52:16.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:16.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:52:16.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:17.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:17.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:17.764 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:52:17.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:52:17.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:17.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:52:17.961 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:52:17.953+0000 7fb97c308540 -1 osd.0 0 log_to_monitors true 2024-09-18T15:52:18.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:19.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:19.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:19.138 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:52:19.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:52:19.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:19.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:52:19.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:20.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:20.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:20.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:52:20.462 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:52:20.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:20.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:52:20.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:21.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:21.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:21.850 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:52:21.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:52:21.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:21.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:52:22.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:23.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:23.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:23.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:52:23.168 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:52:23.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:23.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:52:23.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:24.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:24.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:24.490 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:52:24.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:52:24.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:24.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:52:24.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:25.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:25.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:25.836 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:52:25.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:52:25.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:25.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:52:26.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:27.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:27.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:27.157 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:52:27.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:52:27.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:27.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:52:27.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:28.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:28.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:28.471 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:52:28.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:52:28.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:28.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:52:28.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:29.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:29.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:29.840 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:52:29.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:52:29.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:29.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:52:30.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:31.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:31.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:31.162 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:52:31.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:52:31.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:31.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:52:31.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:32.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:32.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:32.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:52:32.746 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:52:32.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:32.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:52:33.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:34.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:34.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:34.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:52:34.565 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:52:34.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:34.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:52:35.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:36.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:36.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:36.080 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:52:36.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:52:36.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:36.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:52:36.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:37.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:37.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:37.427 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:52:37.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:52:37.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:37.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:52:37.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:38.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:38.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:38.773 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:52:38.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:52:38.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:38.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:52:39.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:40.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:40.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:40.645 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:52:40.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:52:40.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:40.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:52:40.954 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/422906,v1:127.0.0.1:6803/422906] [v2:127.0.0.1:6804/422906,v1:127.0.0.1:6805/422906] exists,up 6b2dfba1-d6b2-4e6a-abac-aeb79442f721 2024-09-18T15:52:40.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:52:40.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:52:40.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:52:40.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5789: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-09-18T15:52:40.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:52:40.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:52:40.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:52:40.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:52:40.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:52:40.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:52:40.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:52:40.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:52:40.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:52:40.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:52:40.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:52:40.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:52:40.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:52:40.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:52:40.958 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:52:40.958 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:52:40.958 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:52:40.960 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:52:40.960 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:52:40.960 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:52:40.960 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:52:40.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:52:40.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:52:40.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:52:40.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:52:40.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:52:40.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:52:40.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:52:40.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:52:40.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:52:40.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:52:40.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:52:40.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-09-18T15:52:40.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:52:40.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:52:40.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bc9593df-548b-463c-83aa-728ba8699eb7 2024-09-18T15:52:40.962 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 bc9593df-548b-463c-83aa-728ba8699eb7 2024-09-18T15:52:40.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 bc9593df-548b-463c-83aa-728ba8699eb7' 2024-09-18T15:52:40.963 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:52:40.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBI9+pmjXOKOhAAKu5oEax+5BxmYlJtzgGYRA== 2024-09-18T15:52:40.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBI9+pmjXOKOhAAKu5oEax+5BxmYlJtzgGYRA=="}' 2024-09-18T15:52:40.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bc9593df-548b-463c-83aa-728ba8699eb7 -i td/osd-scrub-repair/1/new.json 2024-09-18T15:52:41.372 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:52:41.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-18T15:52:41.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQBI9+pmjXOKOhAAKu5oEax+5BxmYlJtzgGYRA== --osd-uuid bc9593df-548b-463c-83aa-728ba8699eb7 2024-09-18T15:52:41.429 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:52:41.421+0000 7fe9087f5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:41.433 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:52:41.425+0000 7fe9087f5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:41.434 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:52:41.425+0000 7fe9087f5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:41.436 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:52:41.429+0000 7fe9087f5540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:52:41.436 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:52:41.429+0000 7fe9087f5540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:52:41.436 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:52:41.429+0000 7fe9087f5540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:52:41.493 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:52:41.485+0000 7fe9087f5540 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-18T15:52:47.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-18T15:52:47.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:52:47.676 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T15:52:47.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:52:47.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:52:48.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:52:48.071 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T15:52:48.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:52:48.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:52:48.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-09-18T15:52:48.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:52:48.103 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:52:48.097+0000 7f12558ce540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:48.106 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:52:48.097+0000 7f12558ce540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:48.107 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:52:48.101+0000 7f12558ce540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:48.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:52:48.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:52:48.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:52:48.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:52:48.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:52:48.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:48.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:52:48.375 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:52:48.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:48.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:52:48.650 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:52:48.641+0000 7f12558ce540 -1 Falling back to public interface 2024-09-18T15:52:48.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:49.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:49.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:49.723 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:52:49.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:52:49.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:49.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:52:50.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:51.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:51.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:51.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:52:51.029 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:52:51.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:51.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:52:51.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:51.361 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:52:51.352+0000 7f12558ce540 -1 osd.1 0 log_to_monitors true 2024-09-18T15:52:52.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:52.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:52.349 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:52:52.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:52:52.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:52.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:52:52.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:53.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:53.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:53.657 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:52:53.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:52:53.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:53.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:52:54.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:55.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:55.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:55.004 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:52:55.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:52:55.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:55.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:52:55.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:56.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:56.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:56.333 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:52:56.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:52:56.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:56.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:52:56.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:57.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:57.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:57.653 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:52:57.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:52:57.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:57.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:52:58.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:59.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:59.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:59.107 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:52:59.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:52:59.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:59.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:52:59.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:00.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:00.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:00.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:53:00.425 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:53:00.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:00.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:53:00.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:01.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:01.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:01.742 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:53:01.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:53:01.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:01.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:53:02.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:03.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:03.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:03.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:53:03.177 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:53:03.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:03.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:53:03.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:04.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:04.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:04.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:53:04.684 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:53:04.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:04.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:53:05.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:06.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:06.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:06.256 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:53:06.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:53:06.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:06.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:53:06.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:07.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:07.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:07.609 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:53:07.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:53:07.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:07.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:53:07.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:08.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:08.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:08.999 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:53:09.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:53:09.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:09.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:53:09.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:10.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:10.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:10.383 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:53:10.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:53:10.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:10.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:53:10.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:11.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:11.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:11.727 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:53:11.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:53:11.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:11.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:53:12.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:13.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:13.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:13.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:53:13.051 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:53:13.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:13.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:53:13.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:14.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:14.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:14.705 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:53:14.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:53:14.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:14.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:53:15.037 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/424686,v1:127.0.0.1:6811/424686] [v2:127.0.0.1:6812/424686,v1:127.0.0.1:6813/424686] exists,up bc9593df-548b-463c-83aa-728ba8699eb7 2024-09-18T15:53:15.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:53:15.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:53:15.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:53:15.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5790: TEST_periodic_scrub_replicated: create_rbd_pool 2024-09-18T15:53:15.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T15:53:15.354 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' does not exist 2024-09-18T15:53:15.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T15:53:15.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T15:53:15.803 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' created 2024-09-18T15:53:15.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:53:16.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T15:53:18.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5791: TEST_periodic_scrub_replicated: wait_for_clean 2024-09-18T15:53:18.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:53:18.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:53:18.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:53:18.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:53:18.670 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:53:18.671 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:53:18.671 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:53:18.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:53:18.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:53:18.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:53:18.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:53:18.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:53:18.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:53:18.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:53:18.914 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:53:19.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:53:19.240 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:53:19.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:53:19.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:53:19.240 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:53:19.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836494 2024-09-18T15:53:19.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836494 2024-09-18T15:53:19.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836494' 2024-09-18T15:53:19.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:53:19.409 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:53:19.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672963 2024-09-18T15:53:19.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672963 2024-09-18T15:53:19.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836494 1-42949672963' 2024-09-18T15:53:19.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:53:19.550 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836494 2024-09-18T15:53:19.550 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:53:19.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:53:19.554 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836494 2024-09-18T15:53:19.555 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:53:19.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836494 2024-09-18T15:53:19.557 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836494 2024-09-18T15:53:19.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-09-18T15:53:19.557 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:53:19.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836493 -lt 21474836494 2024-09-18T15:53:19.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:53:20.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:53:20.876 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:53:21.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836493 -lt 21474836494 2024-09-18T15:53:21.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:53:22.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-18T15:53:22.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:53:22.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836494 -lt 21474836494 2024-09-18T15:53:22.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:53:22.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672963 2024-09-18T15:53:22.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:53:22.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:53:22.517 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672963 2024-09-18T15:53:22.517 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:53:22.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672963 2024-09-18T15:53:22.519 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672963 2024-09-18T15:53:22.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2024-09-18T15:53:22.520 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:53:22.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672964 -lt 42949672963 2024-09-18T15:53:22.860 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:53:22.861 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:53:22.861 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:53:23.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T15:53:23.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:53:23.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:53:23.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:53:23.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:53:23.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:53:23.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:53:23.264 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:53:23.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T15:53:23.602 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:53:23.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:53:23.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:53:23.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T15:53:23.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:53:23.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:53:23.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5793: TEST_periodic_scrub_replicated: create_pool psr_pool 1 1 2024-09-18T15:53:23.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool 1 1 2024-09-18T15:53:24.467 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'psr_pool' created 2024-09-18T15:53:24.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:53:25.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: wait_for_clean 2024-09-18T15:53:25.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:53:25.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:53:25.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:53:25.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:53:25.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:53:25.484 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:53:25.484 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:53:25.484 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:53:25.485 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:53:25.485 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:53:25.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:53:25.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:53:25.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:53:25.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:53:25.725 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:53:26.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:53:26.034 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:53:26.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:53:26.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:53:26.035 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:53:26.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836496 2024-09-18T15:53:26.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836496 2024-09-18T15:53:26.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836496' 2024-09-18T15:53:26.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:53:26.202 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:53:26.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672966 2024-09-18T15:53:26.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672966 2024-09-18T15:53:26.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836496 1-42949672966' 2024-09-18T15:53:26.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:53:26.343 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836496 2024-09-18T15:53:26.343 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:53:26.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:53:26.347 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836496 2024-09-18T15:53:26.348 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:53:26.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836496 2024-09-18T15:53:26.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-09-18T15:53:26.350 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836496 2024-09-18T15:53:26.350 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:53:26.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836495 -lt 21474836496 2024-09-18T15:53:26.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:53:27.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:53:27.665 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:53:28.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836497 -lt 21474836496 2024-09-18T15:53:28.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:53:28.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672966 2024-09-18T15:53:28.032 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:53:28.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:53:28.036 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672966 2024-09-18T15:53:28.036 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:53:28.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672966 2024-09-18T15:53:28.039 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672966 2024-09-18T15:53:28.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-09-18T15:53:28.040 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:53:28.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672966 -lt 42949672966 2024-09-18T15:53:28.401 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:53:28.402 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:53:28.402 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:53:28.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:53:28.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:53:28.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:53:28.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:53:28.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:53:28.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:53:28.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:53:28.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:53:29.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:53:29.098 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:53:29.098 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:53:29.098 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:53:29.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:53:29.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:53:29.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:53:29.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5796: TEST_periodic_scrub_replicated: local osd=0 2024-09-18T15:53:29.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5797: TEST_periodic_scrub_replicated: add_something td/osd-scrub-repair psr_pool POBJ scrub 2024-09-18T15:53:29.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T15:53:29.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=psr_pool 2024-09-18T15:53:29.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=POBJ 2024-09-18T15:53:29.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=scrub 2024-09-18T15:53:29.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' scrub = noscrub ']' 2024-09-18T15:53:29.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:80: add_something: ceph osd unset noscrub 2024-09-18T15:53:29.930 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is unset 2024-09-18T15:53:29.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:81: add_something: ceph osd unset nodeep-scrub 2024-09-18T15:53:30.406 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is unset 2024-09-18T15:53:30.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T15:53:30.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T15:53:30.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool psr_pool put POBJ td/osd-scrub-repair/ORIGINAL 2024-09-18T15:53:30.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5798: TEST_periodic_scrub_replicated: get_primary psr_pool POBJ 2024-09-18T15:53:30.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=psr_pool 2024-09-18T15:53:30.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=POBJ 2024-09-18T15:53:30.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map psr_pool POBJ 2024-09-18T15:53:30.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:53:30.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5798: TEST_periodic_scrub_replicated: local primary=0 2024-09-18T15:53:30.984 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5799: TEST_periodic_scrub_replicated: get_pg psr_pool POBJ 2024-09-18T15:53:30.984 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=psr_pool 2024-09-18T15:53:30.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=POBJ 2024-09-18T15:53:30.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map psr_pool POBJ 2024-09-18T15:53:30.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:53:31.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5799: TEST_periodic_scrub_replicated: local pg=2.0 2024-09-18T15:53:31.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5802: TEST_periodic_scrub_replicated: local payload=UVWXYZ 2024-09-18T15:53:31.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5803: TEST_periodic_scrub_replicated: echo UVWXYZ 2024-09-18T15:53:31.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5805: TEST_periodic_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:53:31.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T15:53:31.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:53:31.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T15:53:31.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:53:31.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:53:31.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T15:53:31.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:53:31.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T15:53:31.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:53:31.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:53:31.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:53:31.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:53:31.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:53:31.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:53:31.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:53:31.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:53:31.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:53:31.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:53:31.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:53:31.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:53:31.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:53:31.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:53:31.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-09-18T15:53:33.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-09-18T15:53:33.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:53:33.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:53:33.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:53:33.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:53:33.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:53:33.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:53:33.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:53:33.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:53:33.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:53:33.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:53:33.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:53:33.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:53:33.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:53:33.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:53:33.760 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:53:33.760 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:53:33.760 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:53:33.760 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:53:33.760 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:53:33.760 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:53:33.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:53:33.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:53:33.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:53:33.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:53:33.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:53:33.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:53:33.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:53:33.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:53:33.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:53:33.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:53:33.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-09-18T15:53:33.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:53:33.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:53:33.764 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T15:53:33.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-09-18T15:53:33.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:53:33.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:53:33.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:53:33.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:53:33.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:53:33.798 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:53:33.788+0000 7f714d2cb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:53:33.802 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:53:33.792+0000 7f714d2cb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:53:33.803 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:53:33.796+0000 7f714d2cb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:53:34.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:53:34.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:53:34.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:53:34.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:53:34.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:53:34.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:34.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:53:34.080 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T15:53:34.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:34.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:53:34.337 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:53:34.328+0000 7f714d2cb540 -1 Falling back to public interface 2024-09-18T15:53:34.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:35.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:35.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:35.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:53:35.389 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:53:35.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:35.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:53:35.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:36.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:36.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:36.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:53:36.705 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T15:53:36.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:36.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:53:37.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:37.058 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:53:37.048+0000 7f714d2cb540 -1 osd.0 25 log_to_monitors true 2024-09-18T15:53:38.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:38.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:38.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:53:38.021 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T15:53:38.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:38.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:53:38.378 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 29 up_thru 29 down_at 26 last_clean_interval [5,25) [v2:127.0.0.1:6802/426950,v1:127.0.0.1:6803/426950] [v2:127.0.0.1:6804/426950,v1:127.0.0.1:6805/426950] exists,up 6b2dfba1-d6b2-4e6a-abac-aeb79442f721 2024-09-18T15:53:38.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:53:38.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:53:38.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:53:38.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:53:38.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:53:38.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:53:38.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:53:38.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:53:38.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:53:38.379 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:53:38.380 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:53:38.380 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:53:38.380 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:53:38.380 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:53:38.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:53:38.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:53:38.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:53:38.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:53:38.616 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:53:38.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:53:38.967 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:53:38.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:53:38.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:53:38.968 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:53:39.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051586 2024-09-18T15:53:39.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051586 2024-09-18T15:53:39.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051586' 2024-09-18T15:53:39.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:53:39.108 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:53:39.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672969 2024-09-18T15:53:39.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672969 2024-09-18T15:53:39.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051586 1-42949672969' 2024-09-18T15:53:39.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:53:39.237 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-124554051586 2024-09-18T15:53:39.238 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:53:39.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:53:39.242 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-124554051586 2024-09-18T15:53:39.242 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:53:39.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051586 2024-09-18T15:53:39.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-09-18T15:53:39.244 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 124554051586 2024-09-18T15:53:39.245 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:53:39.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051586 -lt 124554051586 2024-09-18T15:53:39.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:53:39.554 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672969 2024-09-18T15:53:39.555 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:53:39.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:53:39.558 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672969 2024-09-18T15:53:39.558 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:53:39.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672969 2024-09-18T15:53:39.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-09-18T15:53:39.561 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 42949672969 2024-09-18T15:53:39.561 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:53:39.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672968 -lt 42949672969 2024-09-18T15:53:39.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:53:40.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:53:40.909 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:53:41.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672968 -lt 42949672969 2024-09-18T15:53:41.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:53:42.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-18T15:53:42.252 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:53:42.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672970 -lt 42949672969 2024-09-18T15:53:42.583 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:53:42.584 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:53:42.584 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:53:42.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T15:53:42.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:53:42.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:53:42.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:53:42.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:53:42.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:53:42.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:53:42.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:53:43.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:53:43.335 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:53:43.335 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:53:43.336 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:53:43.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T15:53:43.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:53:43.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:53:43.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5808: TEST_periodic_scrub_replicated: set -o pipefail 2024-09-18T15:53:43.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5809: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-09-18T15:53:43.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5809: TEST_periodic_scrub_replicated: jq . 2024-09-18T15:53:43.733 INFO:tasks.workunit.client.0.smithi046.stderr:No scrub information available for pg 2.0 2024-09-18T15:53:43.733 INFO:tasks.workunit.client.0.smithi046.stderr:error 2: (2) No such file or directory 2024-09-18T15:53:43.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5810: TEST_periodic_scrub_replicated: set +o pipefail 2024-09-18T15:53:43.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5812: TEST_periodic_scrub_replicated: pg_deep_scrub 2.0 2024-09-18T15:53:43.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: pg_deep_scrub: local pgid=2.0 2024-09-18T15:53:43.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: pg_deep_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-09-18T15:53:43.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:53:43.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:53:43.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:53:43.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-09-18T15:53:44.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: pg_deep_scrub: local last_scrub=2024-09-18T15:53:24.375694+0000 2024-09-18T15:53:44.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: pg_deep_scrub: ceph pg deep-scrub 2.0 2024-09-18T15:53:44.314 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 2.0 on osd.0 to deep-scrub 2024-09-18T15:53:44.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: pg_deep_scrub: wait_for_scrub 2.0 2024-09-18T15:53:24.375694+0000 last_deep_scrub_stamp 2024-09-18T15:53:44.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=2.0 2024-09-18T15:53:44.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:53:24.375694+0000 2024-09-18T15:53:44.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-09-18T15:53:44.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:53:44.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:53:44.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-09-18T15:53:44.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:53:44.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:53:44.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:53:44.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-09-18T15:53:44.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:53:24.375694+0000 '>' 2024-09-18T15:53:24.375694+0000 2024-09-18T15:53:44.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:53:45.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:53:45.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:53:45.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-09-18T15:53:45.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:53:45.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:53:45.629 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:53:45.629 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-09-18T15:53:45.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:53:24.375694+0000 '>' 2024-09-18T15:53:24.375694+0000 2024-09-18T15:53:45.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:53:46.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:53:46.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:53:46.949 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-09-18T15:53:46.949 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:53:46.949 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:53:46.949 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:53:46.950 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-09-18T15:53:47.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:53:24.375694+0000 '>' 2024-09-18T15:53:24.375694+0000 2024-09-18T15:53:47.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:53:48.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:53:48.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:53:48.266 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-09-18T15:53:48.266 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:53:48.266 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:53:48.266 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:53:48.267 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-09-18T15:53:48.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:53:24.375694+0000 '>' 2024-09-18T15:53:24.375694+0000 2024-09-18T15:53:48.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:53:49.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:53:49.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:53:49.583 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-09-18T15:53:49.583 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:53:49.584 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T15:53:49.584 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:53:49.584 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-09-18T15:53:49.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:53:45.012189+0000 '>' 2024-09-18T15:53:24.375694+0000 2024-09-18T15:53:49.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:53:49.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5815: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-09-18T15:53:49.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5815: TEST_periodic_scrub_replicated: jq . 2024-09-18T15:53:49.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5815: TEST_periodic_scrub_replicated: grep -q POBJ 2024-09-18T15:53:49.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5817: TEST_periodic_scrub_replicated: flush_pg_stats 2024-09-18T15:53:49.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:53:49.945 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:53:50.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:53:50.242 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:53:50.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:53:50.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:53:50.243 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:53:50.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051589 2024-09-18T15:53:50.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051589 2024-09-18T15:53:50.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051589' 2024-09-18T15:53:50.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:53:50.415 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:53:50.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672973 2024-09-18T15:53:50.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672973 2024-09-18T15:53:50.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051589 1-42949672973' 2024-09-18T15:53:50.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:53:50.552 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-124554051589 2024-09-18T15:53:50.552 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:53:50.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:53:50.556 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-124554051589 2024-09-18T15:53:50.556 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:53:50.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051589 2024-09-18T15:53:50.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 124554051589' 2024-09-18T15:53:50.559 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 124554051589 2024-09-18T15:53:50.559 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:53:50.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051588 -lt 124554051589 2024-09-18T15:53:50.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:53:51.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:53:51.871 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:53:52.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051589 -lt 124554051589 2024-09-18T15:53:52.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:53:52.227 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672973 2024-09-18T15:53:52.227 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:53:52.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:53:52.230 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672973 2024-09-18T15:53:52.231 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:53:52.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672973 2024-09-18T15:53:52.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-09-18T15:53:52.233 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672973 2024-09-18T15:53:52.234 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:53:52.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672973 -lt 42949672973 2024-09-18T15:53:52.566 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5818: TEST_periodic_scrub_replicated: get_last_scrub_stamp 2.0 2024-09-18T15:53:52.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:53:52.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:53:52.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:53:52.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:53:52.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5818: TEST_periodic_scrub_replicated: local last_scrub=2024-09-18T15:53:45.012189+0000 2024-09-18T15:53:52.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5820: TEST_periodic_scrub_replicated: ceph tell 2.0 scrub 2024-09-18T15:53:53.051 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T15:53:53.052 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T15:53:53.052 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T15:52:13.048504+0000" 2024-09-18T15:53:53.052 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T15:53:53.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5822: TEST_periodic_scrub_replicated: wait_for_scrub 2.0 2024-09-18T15:53:45.012189+0000 2024-09-18T15:53:53.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=2.0 2024-09-18T15:53:53.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:53:45.012189+0000 2024-09-18T15:53:53.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:53:53.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:53:53.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:53:53.063 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:53:53.063 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:53:53.063 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:53:53.064 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:53:53.064 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:53:53.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:53:45.012189+0000 '>' 2024-09-18T15:53:45.012189+0000 2024-09-18T15:53:53.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:53:54.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:53:54.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:53:54.362 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:53:54.362 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:53:54.362 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:53:54.362 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:53:54.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:53:54.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:53:45.012189+0000 '>' 2024-09-18T15:53:45.012189+0000 2024-09-18T15:53:54.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:53:55.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:53:55.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:53:55.678 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:53:55.679 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:53:55.679 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:53:55.679 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:53:55.679 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:53:55.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:53:45.012189+0000 '>' 2024-09-18T15:53:45.012189+0000 2024-09-18T15:53:55.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:53:57.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:53:57.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:53:57.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:53:57.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:53:57.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:53:57.003 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:53:57.003 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:53:57.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:53:45.012189+0000 '>' 2024-09-18T15:53:45.012189+0000 2024-09-18T15:53:57.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:53:58.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:53:58.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:53:58.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:53:58.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:53:58.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:53:58.320 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:53:58.320 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:53:58.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:53:45.012189+0000 '>' 2024-09-18T15:53:45.012189+0000 2024-09-18T15:53:58.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:53:59.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:53:59.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:53:59.631 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:53:59.631 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:53:59.632 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:53:59.632 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:53:59.632 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:53:59.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:53:53.094690+0000 '>' 2024-09-18T15:53:45.012189+0000 2024-09-18T15:53:59.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:53:59.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5825: TEST_periodic_scrub_replicated: grep -q 'Deep scrub errors, upgrading scrub to deep-scrub' td/osd-scrub-repair/osd.0.log 2024-09-18T15:53:59.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5828: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-09-18T15:53:59.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5828: TEST_periodic_scrub_replicated: jq . 2024-09-18T15:53:59.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5828: TEST_periodic_scrub_replicated: grep -q POBJ 2024-09-18T15:54:00.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5831: TEST_periodic_scrub_replicated: ceph osd set nodeep-scrub 2024-09-18T15:54:00.466 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T15:54:00.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5833: TEST_periodic_scrub_replicated: ceph tell osd.0 get_latest_osdmap 2024-09-18T15:54:00.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5834: TEST_periodic_scrub_replicated: flush_pg_stats 2024-09-18T15:54:00.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:54:00.653 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:54:00.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:54:00.988 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:54:00.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:54:00.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:54:00.990 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:54:01.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051592 2024-09-18T15:54:01.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051592 2024-09-18T15:54:01.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051592' 2024-09-18T15:54:01.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:54:01.150 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:54:01.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672976 2024-09-18T15:54:01.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672976 2024-09-18T15:54:01.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051592 1-42949672976' 2024-09-18T15:54:01.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:54:01.320 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:54:01.320 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-124554051592 2024-09-18T15:54:01.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:54:01.323 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:54:01.324 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-124554051592 2024-09-18T15:54:01.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051592 2024-09-18T15:54:01.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 124554051592' 2024-09-18T15:54:01.325 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 124554051592 2024-09-18T15:54:01.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:54:01.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051592 -lt 124554051592 2024-09-18T15:54:01.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:54:01.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672976 2024-09-18T15:54:01.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:54:01.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:54:01.640 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672976 2024-09-18T15:54:01.640 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:54:01.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672976 2024-09-18T15:54:01.642 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672976 2024-09-18T15:54:01.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-18T15:54:01.642 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:54:01.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672975 -lt 42949672976 2024-09-18T15:54:01.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:54:02.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:54:02.992 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:54:03.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672976 -lt 42949672976 2024-09-18T15:54:03.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5835: TEST_periodic_scrub_replicated: sleep 5 2024-09-18T15:54:08.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: ceph tell 2.0 scrub 2024-09-18T15:54:08.447 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T15:54:08.448 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T15:54:08.448 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T15:52:28.444357+0000" 2024-09-18T15:54:08.448 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T15:54:08.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5841: TEST_periodic_scrub_replicated: local found=false 2024-09-18T15:54:08.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5842: TEST_periodic_scrub_replicated: seq 14 -1 0 2024-09-18T15:54:08.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5842: TEST_periodic_scrub_replicated: for i in $(seq 14 -1 0) 2024-09-18T15:54:08.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5844: TEST_periodic_scrub_replicated: sleep 1 2024-09-18T15:54:09.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5845: TEST_periodic_scrub_replicated: grep -q 'Regular scrub skipped due to deep-scrub errors and nodeep-scrub set' td/osd-scrub-repair/osd.0.log 2024-09-18T15:54:09.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5845: TEST_periodic_scrub_replicated: found=true 2024-09-18T15:54:09.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5845: TEST_periodic_scrub_replicated: break 2024-09-18T15:54:09.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5848: TEST_periodic_scrub_replicated: test true = true 2024-09-18T15:54:09.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5851: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-09-18T15:54:09.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5851: TEST_periodic_scrub_replicated: jq . 2024-09-18T15:54:09.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5851: TEST_periodic_scrub_replicated: grep -q POBJ 2024-09-18T15:54:09.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5853: TEST_periodic_scrub_replicated: flush_pg_stats 2024-09-18T15:54:09.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:54:09.522 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:54:09.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:54:09.832 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:54:09.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:54:09.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:54:09.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:54:10.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051595 2024-09-18T15:54:10.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051595 2024-09-18T15:54:10.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051595' 2024-09-18T15:54:10.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:54:10.013 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:54:10.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672979 2024-09-18T15:54:10.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672979 2024-09-18T15:54:10.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051595 1-42949672979' 2024-09-18T15:54:10.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:54:10.158 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-124554051595 2024-09-18T15:54:10.159 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:54:10.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:54:10.163 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-124554051595 2024-09-18T15:54:10.163 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:54:10.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051595 2024-09-18T15:54:10.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 124554051595' 2024-09-18T15:54:10.166 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 124554051595 2024-09-18T15:54:10.166 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:54:10.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051594 -lt 124554051595 2024-09-18T15:54:10.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:54:11.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:54:11.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:54:11.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051595 -lt 124554051595 2024-09-18T15:54:11.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:54:11.790 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672979 2024-09-18T15:54:11.790 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:54:11.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:54:11.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672979 2024-09-18T15:54:11.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:54:11.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672979 2024-09-18T15:54:11.796 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672979 2024-09-18T15:54:11.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-18T15:54:11.797 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:54:12.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672979 -lt 42949672979 2024-09-18T15:54:12.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5855: TEST_periodic_scrub_replicated: pg_scrub 2.0 2024-09-18T15:54:12.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: pg_scrub: local pgid=2.0 2024-09-18T15:54:12.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: pg_scrub: get_last_scrub_stamp 2.0 2024-09-18T15:54:12.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:54:12.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:54:12.172 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:54:12.172 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:54:12.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: pg_scrub: local last_scrub=2024-09-11T15:52:28.444357+0000 2024-09-18T15:54:12.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_scrub: ceph pg scrub 2.0 2024-09-18T15:54:12.808 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 2.0 on osd.0 to scrub 2024-09-18T15:54:12.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_scrub: wait_for_scrub 2.0 2024-09-11T15:52:28.444357+0000 2024-09-18T15:54:12.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=2.0 2024-09-18T15:54:12.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-11T15:52:28.444357+0000 2024-09-18T15:54:12.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:54:12.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:54:12.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:54:12.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:54:12.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:54:12.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:54:12.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:54:12.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:54:13.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T15:52:28.444357+0000 '>' 2024-09-11T15:52:28.444357+0000 2024-09-18T15:54:13.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:54:14.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:54:14.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:54:14.169 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:54:14.169 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:54:14.170 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:54:14.170 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:54:14.170 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:54:14.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T15:52:28.444357+0000 '>' 2024-09-11T15:52:28.444357+0000 2024-09-18T15:54:14.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:54:15.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:54:15.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:54:15.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:54:15.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:54:15.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:54:15.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:54:15.496 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:54:15.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T15:52:28.444357+0000 '>' 2024-09-11T15:52:28.444357+0000 2024-09-18T15:54:15.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:54:16.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:54:16.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:54:16.810 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:54:16.810 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:54:16.810 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:54:16.811 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:54:16.811 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:54:17.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T15:52:28.444357+0000 '>' 2024-09-11T15:52:28.444357+0000 2024-09-18T15:54:17.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:54:18.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:54:18.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:54:18.127 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:54:18.127 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:54:18.127 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:54:18.128 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:54:18.128 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:54:18.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T15:52:28.444357+0000 '>' 2024-09-11T15:52:28.444357+0000 2024-09-18T15:54:18.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:54:19.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:54:19.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:54:19.444 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-18T15:54:19.444 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-09-18T15:54:19.444 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:54:19.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:54:19.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-18T15:54:19.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:54:12.846978+0000 '>' 2024-09-11T15:52:28.444357+0000 2024-09-18T15:54:19.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:54:19.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5856: TEST_periodic_scrub_replicated: grep -q 'Regular scrub request, deep-scrub details will be lost' td/osd-scrub-repair/osd.0.log 2024-09-18T15:54:19.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5859: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-09-18T15:54:19.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5859: TEST_periodic_scrub_replicated: grep -qv POBJ 2024-09-18T15:54:19.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5859: TEST_periodic_scrub_replicated: jq . 2024-09-18T15:54:19.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T15:54:19.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:54:19.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:54:19.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:54:19.800 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:54:19.800 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:54:19.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:54:19.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:54:19.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:54:19.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:54:19.919 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:54:19.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:54:19.922 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:54:19.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:54:19.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:54:19.926 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:54:19.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:54:19.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:54:19.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:54:19.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:54:19.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:54:19.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:54:19.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:54:19.949 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:54:19.949 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:54:19.949 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:54:19.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:54:19.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:54:19.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:54:19.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T15:54:19.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T15:54:19.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T15:54:19.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T15:54:19.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:54:19.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:54:19.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:54:19.952 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:54:19.953 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:54:19.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:54:19.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:54:19.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:54:19.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:54:19.959 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:54:19.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:54:19.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:54:19.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:54:19.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:54:19.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:54:19.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:54:19.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:54:19.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:54:19.966 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:54:19.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:54:19.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:54:19.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:54:19.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:54:19.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:54:19.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:54:19.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:54:19.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:54:19.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:54:19.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T15:54:19.978 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:54:19.978 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:54:19.978 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:54:19.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T15:54:19.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:54:19.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:54:19.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:54:19.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T15:54:19.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_repair_stats td/osd-scrub-repair 2024-09-18T15:54:19.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:655: TEST_repair_stats: local dir=td/osd-scrub-repair 2024-09-18T15:54:19.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:656: TEST_repair_stats: local poolname=testpool 2024-09-18T15:54:19.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:657: TEST_repair_stats: local OSDS=2 2024-09-18T15:54:19.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:658: TEST_repair_stats: local OBJS=30 2024-09-18T15:54:19.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:660: TEST_repair_stats: local REPAIRS=20 2024-09-18T15:54:19.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:663: TEST_repair_stats: run_mon td/osd-scrub-repair a 2024-09-18T15:54:19.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T15:54:19.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:54:19.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:54:19.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:54:19.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T15:54:19.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-18T15:54:20.366 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:54:20.366 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:54:20.366 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:54:20.367 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:54:20.367 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:54:20.367 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:54:20.367 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:54:20.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T15:54:20.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:54:20.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:54:20.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:54:20.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:54:20.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:54:20.657 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:54:20.658 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:54:20.658 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:54:20.658 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:54:20.659 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:54:20.659 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:54:20.659 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:54:20.660 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:54:20.661 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:54:20.661 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T15:54:20.756 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:54:20.756 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:54:20.756 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:54:20.756 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:54:20.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:54:20.758 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:54:20.758 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:54:20.758 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:54:20.758 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:54:20.758 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:54:20.758 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:54:20.759 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:54:20.759 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:54:20.759 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T15:54:20.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:664: TEST_repair_stats: run_mgr td/osd-scrub-repair x 2024-09-18T15:54:20.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T15:54:20.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:54:20.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:54:20.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:54:20.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T15:54:20.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:54:21.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:54:21.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:54:21.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:54:21.705 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:54:21.706 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:54:21.706 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:54:21.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:54:21.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:54:21.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:54:21.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:54:21.744 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: expr 2 - 1 2024-09-18T15:54:21.746 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: seq 0 1 2024-09-18T15:54:21.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T15:54:21.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:668: TEST_repair_stats: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:54:21.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:54:21.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:54:21.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:54:21.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:54:21.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:54:21.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:54:21.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:54:21.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:54:21.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:54:21.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:54:21.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:54:21.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:54:21.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:54:21.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:54:21.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:54:21.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:54:21.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:54:21.752 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:54:21.752 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:54:21.752 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:54:21.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:54:21.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:54:21.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:54:21.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:54:21.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:54:21.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:54:21.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:54:21.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:54:21.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:54:21.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:54:21.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:54:21.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:54:21.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:54:21.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:54:21.755 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:54:21.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d43f969f-84e7-4f09-9bbc-a63d754181aa 2024-09-18T15:54:21.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d43f969f-84e7-4f09-9bbc-a63d754181aa' 2024-09-18T15:54:21.756 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 d43f969f-84e7-4f09-9bbc-a63d754181aa 2024-09-18T15:54:21.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:54:21.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCt9+pmTzYJLhAAgPp0UuukRbClQrMvAtX2yA== 2024-09-18T15:54:21.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCt9+pmTzYJLhAAgPp0UuukRbClQrMvAtX2yA=="}' 2024-09-18T15:54:21.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d43f969f-84e7-4f09-9bbc-a63d754181aa -i td/osd-scrub-repair/0/new.json 2024-09-18T15:54:22.043 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:54:22.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-18T15:54:22.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCt9+pmTzYJLhAAgPp0UuukRbClQrMvAtX2yA== --osd-uuid d43f969f-84e7-4f09-9bbc-a63d754181aa 2024-09-18T15:54:22.091 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:54:22.087+0000 7f192b580540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:54:22.137 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:54:22.131+0000 7f192b580540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:54:22.138 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:54:22.131+0000 7f192b580540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:54:22.140 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:54:22.135+0000 7f192b580540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:54:22.140 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:54:22.135+0000 7f192b580540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:54:22.140 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:54:22.135+0000 7f192b580540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:54:22.184 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:54:22.179+0000 7f192b580540 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-18T15:54:28.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T15:54:28.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:54:28.386 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T15:54:28.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:54:28.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:54:28.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:54:28.787 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T15:54:28.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:54:28.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:54:28.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:54:28.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:54:28.821 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:54:28.815+0000 7f4713b2a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:54:28.823 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:54:28.819+0000 7f4713b2a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:54:28.825 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:54:28.819+0000 7f4713b2a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:54:29.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:54:29.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:54:29.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:54:29.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:54:29.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:54:29.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:29.131 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:54:29.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:54:29.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:29.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:54:29.381 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:54:29.375+0000 7f4713b2a540 -1 Falling back to public interface 2024-09-18T15:54:29.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:30.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:30.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:30.448 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:54:30.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:54:30.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:30.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:54:30.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:31.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:31.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:31.769 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:54:31.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:54:31.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:31.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:54:32.079 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:54:32.075+0000 7f4713b2a540 -1 osd.0 0 log_to_monitors true 2024-09-18T15:54:32.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:33.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:33.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:33.096 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:54:33.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:54:33.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:33.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:54:33.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:34.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:34.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:34.408 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:54:34.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:54:34.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:34.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:54:34.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:35.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:35.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:35.840 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:54:35.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:54:35.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:35.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:54:36.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:37.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:37.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:37.166 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:54:37.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:54:37.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:37.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:54:37.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:38.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:38.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:38.478 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:54:38.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:54:38.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:38.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:54:38.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:39.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:39.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:39.885 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:54:39.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:54:39.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:39.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:54:40.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:41.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:41.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:41.210 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:54:41.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:54:41.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:41.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:54:41.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:42.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:42.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:42.533 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:54:42.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:54:42.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:42.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:54:42.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:43.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:43.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:43.897 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:54:43.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:54:43.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:43.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:54:44.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:45.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:45.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:45.215 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:54:45.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:54:45.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:45.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:54:45.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:46.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:46.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:46.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:54:46.623 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:54:46.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:46.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:54:47.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:48.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:48.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:48.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:54:48.420 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:54:48.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:48.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:54:48.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:49.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:49.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:49.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:54:49.980 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:54:49.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:49.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:54:50.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:51.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:51.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:51.362 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:54:51.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:54:51.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:51.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:54:51.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:52.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:52.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:52.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:54:52.865 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:54:52.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:52.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:54:53.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:54.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:54.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:54.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:54:54.349 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:54:54.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:54.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:54:54.719 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/430229,v1:127.0.0.1:6803/430229] [v2:127.0.0.1:6804/430229,v1:127.0.0.1:6805/430229] exists,up d43f969f-84e7-4f09-9bbc-a63d754181aa 2024-09-18T15:54:54.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:54:54.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:54:54.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:54:54.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T15:54:54.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:668: TEST_repair_stats: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:54:54.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:54:54.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:54:54.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:54:54.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:54:54.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:54:54.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:54:54.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:54:54.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:54:54.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:54:54.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:54:54.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:54:54.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:54:54.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:54:54.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:54:54.723 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:54:54.723 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:54:54.723 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:54:54.723 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:54:54.725 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:54:54.725 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:54:54.725 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:54:54.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:54:54.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:54:54.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:54:54.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:54:54.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:54:54.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:54:54.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:54:54.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:54:54.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:54:54.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:54:54.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:54:54.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:54:54.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:54:54.727 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:54:54.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f6b9dd03-27e6-4b06-9f97-3904f2180baa 2024-09-18T15:54:54.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f6b9dd03-27e6-4b06-9f97-3904f2180baa' 2024-09-18T15:54:54.729 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 f6b9dd03-27e6-4b06-9f97-3904f2180baa 2024-09-18T15:54:54.730 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:54:54.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDO9+pmWqtjLBAA8IKgsP/q9qu1EVdi4zDY0Q== 2024-09-18T15:54:54.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDO9+pmWqtjLBAA8IKgsP/q9qu1EVdi4zDY0Q=="}' 2024-09-18T15:54:54.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f6b9dd03-27e6-4b06-9f97-3904f2180baa -i td/osd-scrub-repair/1/new.json 2024-09-18T15:54:55.170 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:54:55.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-18T15:54:55.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDO9+pmWqtjLBAA8IKgsP/q9qu1EVdi4zDY0Q== --osd-uuid f6b9dd03-27e6-4b06-9f97-3904f2180baa 2024-09-18T15:54:55.225 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:54:55.219+0000 7f268cac5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:54:55.238 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:54:55.231+0000 7f268cac5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:54:55.240 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:54:55.235+0000 7f268cac5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:54:55.241 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:54:55.235+0000 7f268cac5540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:54:55.241 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:54:55.235+0000 7f268cac5540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:54:55.241 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:54:55.235+0000 7f268cac5540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:54:55.306 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:54:55.299+0000 7f268cac5540 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-18T15:55:01.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-18T15:55:01.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:55:01.790 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T15:55:01.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:55:01.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:55:02.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:55:02.188 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T15:55:02.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:55:02.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:55:02.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:55:02.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:55:02.219 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:55:02.215+0000 7f4ffc43b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:55:02.222 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:55:02.215+0000 7f4ffc43b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:55:02.224 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:55:02.219+0000 7f4ffc43b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:55:02.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:55:02.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:55:02.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:55:02.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:55:02.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:55:02.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:55:02.522 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:55:02.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:55:02.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:55:02.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:55:02.781 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:55:02.775+0000 7f4ffc43b540 -1 Falling back to public interface 2024-09-18T15:55:02.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:55:03.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:55:03.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:55:03.866 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:55:03.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:55:03.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:55:03.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:55:04.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:55:05.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:55:05.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:55:05.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:55:05.181 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:55:05.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:55:05.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:55:05.289 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:55:05.283+0000 7f4ffc43b540 -1 osd.1 0 log_to_monitors true 2024-09-18T15:55:05.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:55:06.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:55:06.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:55:06.513 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:55:06.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:55:06.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:55:06.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:55:06.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:55:07.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:55:07.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:55:07.880 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:55:07.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:55:07.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:55:07.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:55:08.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:55:09.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:55:09.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:55:09.281 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:55:09.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:55:09.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:55:09.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:55:09.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:55:10.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:55:10.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:55:10.597 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:55:10.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:55:10.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:55:10.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:55:10.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:55:11.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:55:11.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:55:11.918 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:55:11.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:55:11.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:55:11.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:55:12.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:55:13.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:55:13.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:55:13.312 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:55:13.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:55:13.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:55:13.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:55:13.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:55:14.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:55:14.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:55:14.628 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:55:14.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:55:14.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:55:14.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:55:15.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:55:16.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:55:16.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:55:16.093 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:55:16.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:55:16.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:55:16.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:55:16.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:55:17.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:55:17.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:55:17.419 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:55:17.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:55:17.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:55:17.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:55:17.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:55:18.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:55:18.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:55:18.737 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:55:18.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:55:18.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:55:18.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:55:19.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:55:20.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:55:20.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:55:20.619 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:55:20.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:55:20.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:55:20.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:55:21.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:55:22.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:55:22.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:55:22.243 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:55:22.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:55:22.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:55:22.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:55:22.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:55:23.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:55:23.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:55:23.636 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:55:23.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:55:23.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:55:23.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:55:23.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:55:24.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:55:24.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:55:24.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:55:24.996 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:55:24.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:55:24.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:55:25.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:55:26.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:55:26.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:55:26.619 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:55:26.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:55:26.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:55:26.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:55:27.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:55:28.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:55:28.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:55:28.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:55:28.259 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:55:28.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:55:28.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:55:28.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:55:29.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:55:29.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:55:29.708 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:55:29.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:55:29.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:55:29.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:55:30.027 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/432011,v1:127.0.0.1:6811/432011] [v2:127.0.0.1:6812/432011,v1:127.0.0.1:6813/432011] exists,up f6b9dd03-27e6-4b06-9f97-3904f2180baa 2024-09-18T15:55:30.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:55:30.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:55:30.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:55:30.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:671: TEST_repair_stats: create_pool testpool 1 1 2024-09-18T15:55:30.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-09-18T15:55:30.534 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'testpool' created 2024-09-18T15:55:30.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:55:31.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_repair_stats: ceph osd pool set testpool size 2 2024-09-18T15:55:32.339 INFO:tasks.workunit.client.0.smithi046.stderr:set pool 1 size to 2 2024-09-18T15:55:32.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:673: TEST_repair_stats: wait_for_clean 2024-09-18T15:55:32.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:55:32.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:55:32.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:55:32.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:55:32.353 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:55:32.354 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:55:32.354 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:55:32.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:55:32.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:55:32.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:55:32.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:55:32.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:55:32.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:55:32.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:55:32.589 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:55:32.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:55:32.954 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:55:32.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:55:32.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:55:32.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:55:33.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836494 2024-09-18T15:55:33.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836494 2024-09-18T15:55:33.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836494' 2024-09-18T15:55:33.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:55:33.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:55:33.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672969 2024-09-18T15:55:33.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672969 2024-09-18T15:55:33.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836494 1-42949672969' 2024-09-18T15:55:33.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:55:33.321 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836494 2024-09-18T15:55:33.321 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:55:33.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:55:33.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:55:33.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836494 2024-09-18T15:55:33.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836494 2024-09-18T15:55:33.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-09-18T15:55:33.327 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836494 2024-09-18T15:55:33.327 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:55:33.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836493 -lt 21474836494 2024-09-18T15:55:33.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:55:34.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:55:34.654 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:55:34.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836493 -lt 21474836494 2024-09-18T15:55:34.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:55:35.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-18T15:55:35.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:55:36.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836494 -lt 21474836494 2024-09-18T15:55:36.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:55:36.300 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672969 2024-09-18T15:55:36.300 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:55:36.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:55:36.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672969 2024-09-18T15:55:36.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:55:36.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672969 2024-09-18T15:55:36.306 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672969 2024-09-18T15:55:36.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-09-18T15:55:36.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:55:36.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672970 -lt 42949672969 2024-09-18T15:55:36.652 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:55:36.653 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:55:36.653 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:55:37.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:55:37.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:55:37.012 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:55:37.012 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:55:37.012 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:55:37.012 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:55:37.013 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:55:37.013 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:55:37.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:55:37.330 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:55:37.330 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:55:37.331 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:55:37.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:55:37.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:55:37.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:55:37.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: local payload=ABCDEF 2024-09-18T15:55:37.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: echo ABCDEF 2024-09-18T15:55:37.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: seq 1 30 2024-09-18T15:55:37.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:37.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:37.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:37.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:38.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:38.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:38.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:38.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:38.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:38.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:38.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:38.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:38.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:38.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:38.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:38.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:38.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:38.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:38.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:38.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:39.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:39.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:39.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:39.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:39.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:39.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:39.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:39.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:39.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:39.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:39.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:39.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:39.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:39.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:40.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:40.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:40.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:40.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:40.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:40.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:40.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:40.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:40.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:40.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:40.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:40.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:40.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:40.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:41.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:41.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:41.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:41.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:41.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:41.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:41.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:41.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:41.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:41.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:41.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-18T15:55:41.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:680: TEST_repair_stats: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:55:41.959 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: get_not_primary testpool obj1 2024-09-18T15:55:41.959 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-09-18T15:55:41.959 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-09-18T15:55:41.960 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-09-18T15:55:41.960 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-09-18T15:55:41.960 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:55:41.961 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-09-18T15:55:41.961 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:55:42.277 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T15:55:42.278 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-09-18T15:55:42.278 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T15:55:42.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: local other=0 2024-09-18T15:55:42.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: get_pg testpool obj1 2024-09-18T15:55:42.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-09-18T15:55:42.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-09-18T15:55:42.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-09-18T15:55:42.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:55:42.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: local pgid=1.0 2024-09-18T15:55:42.971 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:687: TEST_repair_stats: get_primary testpool obj1 2024-09-18T15:55:42.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-09-18T15:55:42.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:55:42.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-09-18T15:55:42.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:55:43.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:687: TEST_repair_stats: local primary=1 2024-09-18T15:55:43.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:689: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:55:43.310 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:55:43.310 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:55:43.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:55:43.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:55:43.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:55:43.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:55:43.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:55:43.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:55:43.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:55:43.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:55:43.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:55:43.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:55:43.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:55:43.945 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: seq 1 20 2024-09-18T15:55:43.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-18T15:55:43.948 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: expr 1 % 2 2024-09-18T15:55:43.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: OSD=1 2024-09-18T15:55:43.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-09-18T15:55:43.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:55:43.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:55:43.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:55:43.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:55:43.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:55:43.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-09-18T15:55:45.323 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:ff7b1f36:::obj1:head# 2024-09-18T15:55:45.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-18T15:55:45.858 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: expr 2 % 2 2024-09-18T15:55:45.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: OSD=0 2024-09-18T15:55:45.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-09-18T15:55:45.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:55:45.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:55:45.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:55:45.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:55:45.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:55:45.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-09-18T15:55:47.252 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:104778fc:::obj2:head# 2024-09-18T15:55:47.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-18T15:55:47.790 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: expr 3 % 2 2024-09-18T15:55:47.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: OSD=1 2024-09-18T15:55:47.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-09-18T15:55:47.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:55:47.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:55:47.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:55:47.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:55:47.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:55:47.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-09-18T15:55:48.726 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:8dd16f86:::obj3:head# 2024-09-18T15:55:49.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-18T15:55:49.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: expr 4 % 2 2024-09-18T15:55:49.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: OSD=0 2024-09-18T15:55:49.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-09-18T15:55:49.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:55:49.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:55:49.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:55:49.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:55:49.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:55:49.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-09-18T15:55:50.338 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:0ee9ae15:::obj4:head# 2024-09-18T15:55:50.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-18T15:55:50.874 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: expr 5 % 2 2024-09-18T15:55:50.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: OSD=1 2024-09-18T15:55:50.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-09-18T15:55:50.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:55:50.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:55:50.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:55:50.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:55:50.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:55:50.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-09-18T15:55:51.812 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:c52c9666:::obj5:head# 2024-09-18T15:55:52.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-18T15:55:52.350 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: expr 6 % 2 2024-09-18T15:55:52.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: OSD=0 2024-09-18T15:55:52.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-09-18T15:55:52.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:55:52.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:55:52.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:55:52.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:55:52.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:55:52.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-09-18T15:55:53.244 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:233a42c1:::obj6:head# 2024-09-18T15:55:53.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-18T15:55:53.790 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: expr 7 % 2 2024-09-18T15:55:53.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: OSD=1 2024-09-18T15:55:53.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-09-18T15:55:53.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:55:53.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:55:53.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:55:53.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:55:53.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:55:53.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-09-18T15:55:54.667 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:9a09113a:::obj7:head# 2024-09-18T15:55:55.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-18T15:55:55.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: expr 8 % 2 2024-09-18T15:55:55.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: OSD=0 2024-09-18T15:55:55.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-09-18T15:55:55.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:55:55.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:55:55.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:55:55.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:55:55.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:55:55.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-09-18T15:55:56.373 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:a557efb1:::obj8:head# 2024-09-18T15:55:56.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-18T15:55:56.909 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: expr 9 % 2 2024-09-18T15:55:56.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: OSD=1 2024-09-18T15:55:56.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-09-18T15:55:56.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:55:56.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:55:56.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:55:56.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:55:56.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:55:56.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-09-18T15:55:57.839 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:fe351e27:::obj9:head# 2024-09-18T15:55:58.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-18T15:55:58.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: expr 10 % 2 2024-09-18T15:55:58.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: OSD=0 2024-09-18T15:55:58.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-09-18T15:55:58.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:55:58.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:55:58.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:55:58.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:55:58.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:55:58.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-09-18T15:55:59.297 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:d1337354:::obj10:head# 2024-09-18T15:55:59.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-18T15:55:59.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: expr 11 % 2 2024-09-18T15:55:59.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: OSD=1 2024-09-18T15:55:59.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-09-18T15:55:59.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:55:59.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:55:59.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:55:59.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:55:59.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:55:59.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-09-18T15:56:00.708 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:eeae2a94:::obj11:head# 2024-09-18T15:56:01.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-18T15:56:01.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: expr 12 % 2 2024-09-18T15:56:01.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: OSD=0 2024-09-18T15:56:01.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-09-18T15:56:01.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:56:01.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:56:01.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:56:01.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:56:01.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:56:01.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-09-18T15:56:02.236 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:af5f95cb:::obj12:head# 2024-09-18T15:56:03.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-18T15:56:03.025 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: expr 13 % 2 2024-09-18T15:56:03.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: OSD=1 2024-09-18T15:56:03.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-09-18T15:56:03.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:56:03.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:56:03.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:56:03.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:56:03.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:56:03.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-09-18T15:56:03.973 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:a61ad63e:::obj13:head# 2024-09-18T15:56:04.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-18T15:56:04.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: expr 14 % 2 2024-09-18T15:56:04.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: OSD=0 2024-09-18T15:56:04.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-09-18T15:56:04.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:56:04.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:56:04.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:56:04.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:56:04.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:56:04.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-09-18T15:56:05.474 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:e0a44829:::obj14:head# 2024-09-18T15:56:06.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-18T15:56:06.009 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: expr 15 % 2 2024-09-18T15:56:06.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: OSD=1 2024-09-18T15:56:06.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-09-18T15:56:06.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:56:06.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:56:06.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:56:06.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:56:06.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:56:06.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-09-18T15:56:06.837 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:ab946124:::obj15:head# 2024-09-18T15:56:07.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-18T15:56:07.381 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: expr 16 % 2 2024-09-18T15:56:07.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: OSD=0 2024-09-18T15:56:07.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-09-18T15:56:07.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:56:07.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:56:07.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:56:07.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:56:07.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:56:07.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-09-18T15:56:08.298 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:461f8b5e:::obj16:head# 2024-09-18T15:56:08.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-18T15:56:08.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: expr 17 % 2 2024-09-18T15:56:08.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: OSD=1 2024-09-18T15:56:08.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-09-18T15:56:08.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:56:08.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:56:08.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:56:08.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:56:08.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:56:08.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-09-18T15:56:09.709 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:24fc6e92:::obj17:head# 2024-09-18T15:56:10.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-18T15:56:10.245 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: expr 18 % 2 2024-09-18T15:56:10.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: OSD=0 2024-09-18T15:56:10.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-09-18T15:56:10.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:56:10.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:56:10.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:56:10.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:56:10.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:56:10.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-09-18T15:56:11.067 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:b9836a99:::obj18:head# 2024-09-18T15:56:11.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-18T15:56:11.854 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: expr 19 % 2 2024-09-18T15:56:11.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: OSD=1 2024-09-18T15:56:11.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-09-18T15:56:11.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:56:11.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:56:11.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:56:11.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:56:11.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:56:11.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-09-18T15:56:12.766 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:00b31b6c:::obj19:head# 2024-09-18T15:56:13.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-18T15:56:13.302 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: expr 20 % 2 2024-09-18T15:56:13.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: OSD=0 2024-09-18T15:56:13.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-09-18T15:56:13.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:56:13.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:56:13.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:56:13.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:56:13.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:56:13.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-09-18T15:56:14.201 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:24b5611b:::obj20:head# 2024-09-18T15:56:14.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:697: TEST_repair_stats: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:56:14.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:56:14.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:56:14.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:56:14.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:56:14.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:56:14.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:56:14.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:56:14.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:56:14.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:56:14.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:56:14.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:56:14.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:56:14.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:56:14.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:56:14.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:56:14.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:56:14.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:56:14.752 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:56:14.752 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:56:14.752 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:56:14.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:56:14.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:56:14.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:56:14.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:56:14.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:56:14.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:56:14.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:56:14.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:56:14.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:56:14.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:56:14.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:56:14.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:56:14.756 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T15:56:14.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:56:14.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:56:14.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:56:14.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:56:14.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:56:14.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:56:14.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:56:14.789 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:56:14.782+0000 7fe755706540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:56:14.794 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:56:14.786+0000 7fe755706540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:56:14.795 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:56:14.790+0000 7fe755706540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:56:15.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:56:15.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:56:15.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:56:15.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:56:15.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:56:15.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:15.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:56:15.072 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:56:15.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:56:15.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:15.336 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:56:15.330+0000 7fe755706540 -1 Falling back to public interface 2024-09-18T15:56:15.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:16.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:16.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:16.443 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:56:16.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:56:16.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:16.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:56:16.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:17.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:17.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:17.749 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:56:17.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:56:17.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:17.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:56:18.030 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:56:18.022+0000 7fe755706540 -1 osd.1 17 log_to_monitors true 2024-09-18T15:56:18.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:19.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:19.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:19.080 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:56:19.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:56:19.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:19.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:56:19.447 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 20 up_thru 20 down_at 18 last_clean_interval [10,17) [v2:127.0.0.1:6802/441814,v1:127.0.0.1:6803/441814] [v2:127.0.0.1:6804/441814,v1:127.0.0.1:6805/441814] exists,up f6b9dd03-27e6-4b06-9f97-3904f2180baa 2024-09-18T15:56:19.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:56:19.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:56:19.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:56:19.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:698: TEST_repair_stats: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:56:19.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:56:19.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:56:19.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:56:19.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:56:19.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:56:19.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:56:19.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:56:19.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:56:19.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:56:19.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:56:19.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:56:19.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:56:19.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:56:19.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:56:19.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:56:19.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:56:19.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:56:19.451 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:56:19.451 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:56:19.451 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:56:19.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:56:19.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:56:19.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:56:19.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:56:19.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:56:19.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:56:19.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:56:19.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:56:19.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:56:19.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:56:19.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:56:19.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:56:19.454 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T15:56:19.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:56:19.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:56:19.456 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:56:19.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:56:19.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:56:19.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:56:19.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:56:19.484 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:56:19.478+0000 7f1749d7a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:56:19.520 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:56:19.514+0000 7f1749d7a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:56:19.521 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:56:19.514+0000 7f1749d7a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:56:19.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:56:19.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:56:19.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:56:19.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:56:19.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:56:19.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:19.800 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:56:19.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:56:19.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:19.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:20.061 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:56:20.054+0000 7f1749d7a540 -1 Falling back to public interface 2024-09-18T15:56:20.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:21.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:21.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:21.134 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:56:21.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:56:21.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:21.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:21.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:22.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:22.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:22.446 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:56:22.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:56:22.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:22.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:22.623 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:56:22.618+0000 7f1749d7a540 -1 osd.0 16 log_to_monitors true 2024-09-18T15:56:22.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:23.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:23.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:23.827 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:56:23.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:56:23.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:23.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:24.143 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 17 last_clean_interval [5,16) [v2:127.0.0.1:6810/442393,v1:127.0.0.1:6811/442393] [v2:127.0.0.1:6812/442393,v1:127.0.0.1:6813/442393] exists,up d43f969f-84e7-4f09-9bbc-a63d754181aa 2024-09-18T15:56:24.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:56:24.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:56:24.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:56:24.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:699: TEST_repair_stats: wait_for_clean 2024-09-18T15:56:24.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:56:24.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:56:24.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:56:24.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:56:24.145 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:56:24.146 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:56:24.146 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:56:24.147 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:56:24.147 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:56:24.147 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:56:24.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:56:24.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:56:24.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:56:24.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:56:24.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:56:24.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:56:24.697 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:56:24.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:56:24.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:56:24.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:56:24.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=98784247810 2024-09-18T15:56:24.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 98784247810 2024-09-18T15:56:24.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-98784247810' 2024-09-18T15:56:24.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:56:24.872 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:56:25.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345923 2024-09-18T15:56:25.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345923 2024-09-18T15:56:25.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-98784247810 1-85899345923' 2024-09-18T15:56:25.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:56:25.012 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-98784247810 2024-09-18T15:56:25.012 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:56:25.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:56:25.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-98784247810 2024-09-18T15:56:25.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:56:25.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=98784247810 2024-09-18T15:56:25.019 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 98784247810 2024-09-18T15:56:25.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 98784247810' 2024-09-18T15:56:25.020 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:56:25.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 98784247810 -lt 98784247810 2024-09-18T15:56:25.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:56:25.329 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-85899345923 2024-09-18T15:56:25.333 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:56:25.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:56:25.333 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-85899345923 2024-09-18T15:56:25.334 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:56:25.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345923 2024-09-18T15:56:25.336 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 85899345923 2024-09-18T15:56:25.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 85899345923' 2024-09-18T15:56:25.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:56:25.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345923 -lt 85899345923 2024-09-18T15:56:25.683 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:56:25.684 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:56:25.684 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:56:26.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:56:26.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:56:26.090 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:56:26.090 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:56:26.090 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:56:26.090 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:56:26.090 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:56:26.091 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:56:26.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:56:26.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:56:26.441 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:56:26.441 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:56:26.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:56:26.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:56:26.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:56:26.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:701: TEST_repair_stats: repair 1.0 2024-09-18T15:56:26.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=1.0 2024-09-18T15:56:26.809 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 1.0 2024-09-18T15:56:26.809 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:56:26.809 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:56:26.809 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:56:26.810 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:56:27.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T15:55:30.443244+0000 2024-09-18T15:56:27.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 1.0 2024-09-18T15:56:27.426 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 1.0 on osd.1 to repair 2024-09-18T15:56:27.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 1.0 2024-09-18T15:55:30.443244+0000 2024-09-18T15:56:27.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-18T15:56:27.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:55:30.443244+0000 2024-09-18T15:56:27.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:56:27.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:56:27.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:56:27.443 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T15:56:27.443 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:56:27.443 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:56:27.443 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:56:27.443 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:56:27.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:55:30.443244+0000 '>' 2024-09-18T15:55:30.443244+0000 2024-09-18T15:56:27.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:56:28.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:56:28.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:56:28.739 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T15:56:28.739 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:56:28.739 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:56:28.739 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:56:28.740 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:56:29.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:55:30.443244+0000 '>' 2024-09-18T15:55:30.443244+0000 2024-09-18T15:56:29.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:56:30.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:56:30.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:56:30.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T15:56:30.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:56:30.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:56:30.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:56:30.055 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:56:30.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:56:28.104324+0000 '>' 2024-09-18T15:55:30.443244+0000 2024-09-18T15:56:30.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:56:30.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:702: TEST_repair_stats: wait_for_clean 2024-09-18T15:56:30.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:56:30.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:56:30.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:56:30.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:56:30.367 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:56:30.367 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:56:30.367 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:56:30.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:56:30.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:56:30.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:56:30.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:56:30.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:56:30.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:56:30.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:56:30.614 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:56:30.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:56:30.937 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:56:30.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:56:30.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:56:30.938 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:56:31.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=98784247812 2024-09-18T15:56:31.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 98784247812 2024-09-18T15:56:31.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-98784247812' 2024-09-18T15:56:31.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:56:31.107 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:56:31.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345925 2024-09-18T15:56:31.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345925 2024-09-18T15:56:31.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-98784247812 1-85899345925' 2024-09-18T15:56:31.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:56:31.258 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-98784247812 2024-09-18T15:56:31.258 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:56:31.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:56:31.259 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-98784247812 2024-09-18T15:56:31.259 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:56:31.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=98784247812 2024-09-18T15:56:31.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 98784247812' 2024-09-18T15:56:31.262 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 98784247812 2024-09-18T15:56:31.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:56:31.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 98784247811 -lt 98784247812 2024-09-18T15:56:31.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:56:32.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:56:32.569 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:56:32.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 98784247811 -lt 98784247812 2024-09-18T15:56:32.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:56:33.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-18T15:56:33.896 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:56:34.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 98784247813 -lt 98784247812 2024-09-18T15:56:34.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:56:34.214 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-85899345925 2024-09-18T15:56:34.214 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:56:34.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:56:34.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-85899345925 2024-09-18T15:56:34.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:56:34.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345925 2024-09-18T15:56:34.221 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 85899345925 2024-09-18T15:56:34.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 85899345925' 2024-09-18T15:56:34.222 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:56:34.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345925 -lt 85899345925 2024-09-18T15:56:34.562 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:56:34.563 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:56:34.563 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:56:34.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:56:34.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:56:34.977 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:56:34.977 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:56:34.977 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:56:34.977 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:56:34.978 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:56:34.978 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:56:35.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:56:35.278 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:56:35.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:56:35.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:56:35.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:56:35.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:56:35.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:56:35.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:703: TEST_repair_stats: ceph pg dump pgs 2024-09-18T15:56:35.974 INFO:tasks.workunit.client.0.smithi046.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T15:56:35.974 INFO:tasks.workunit.client.0.smithi046.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-09-18T15:56:28.908491+0000 16'30 24:127 [1,0] 1 [1,0] 1 16'30 2024-09-18T15:56:28.104324+0000 16'30 2024-09-18T15:56:28.104324+0000 0 1 periodic scrub scheduled @ 2024-09-19T15:56:28.104324+0000 20 0 2024-09-18T15:56:35.974 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-09-18T15:56:35.974 INFO:tasks.workunit.client.0.smithi046.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T15:56:35.978 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T15:56:35.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:704: TEST_repair_stats: flush_pg_stats 2024-09-18T15:56:35.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:56:35.988 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:56:36.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:56:36.288 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T15:56:36.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:56:36.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:56:36.289 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:56:36.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=98784247814 2024-09-18T15:56:36.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 98784247814 2024-09-18T15:56:36.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-98784247814' 2024-09-18T15:56:36.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:56:36.476 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:56:36.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345927 2024-09-18T15:56:36.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345927 2024-09-18T15:56:36.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-98784247814 1-85899345927' 2024-09-18T15:56:36.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:56:36.613 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-98784247814 2024-09-18T15:56:36.613 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:56:36.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:56:36.617 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-98784247814 2024-09-18T15:56:36.617 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:56:36.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=98784247814 2024-09-18T15:56:36.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 98784247814' 2024-09-18T15:56:36.620 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 98784247814 2024-09-18T15:56:36.620 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:56:36.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 98784247813 -lt 98784247814 2024-09-18T15:56:36.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:56:37.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:56:37.934 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:56:38.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 98784247814 -lt 98784247814 2024-09-18T15:56:38.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:56:38.252 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-85899345927 2024-09-18T15:56:38.252 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:56:38.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:56:38.255 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-85899345927 2024-09-18T15:56:38.256 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:56:38.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345927 2024-09-18T15:56:38.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 85899345927' 2024-09-18T15:56:38.258 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 85899345927 2024-09-18T15:56:38.259 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:56:38.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345927 -lt 85899345927 2024-09-18T15:56:38.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: ceph pg 1.0 query 2024-09-18T15:56:38.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: jq .info.stats.stat_sum 2024-09-18T15:56:38.780 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T15:56:38.780 INFO:tasks.workunit.client.0.smithi046.stdout: "num_bytes": 210, 2024-09-18T15:56:38.780 INFO:tasks.workunit.client.0.smithi046.stdout: "num_objects": 30, 2024-09-18T15:56:38.781 INFO:tasks.workunit.client.0.smithi046.stdout: "num_object_clones": 0, 2024-09-18T15:56:38.781 INFO:tasks.workunit.client.0.smithi046.stdout: "num_object_copies": 60, 2024-09-18T15:56:38.781 INFO:tasks.workunit.client.0.smithi046.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T15:56:38.781 INFO:tasks.workunit.client.0.smithi046.stdout: "num_objects_missing": 0, 2024-09-18T15:56:38.781 INFO:tasks.workunit.client.0.smithi046.stdout: "num_objects_degraded": 0, 2024-09-18T15:56:38.781 INFO:tasks.workunit.client.0.smithi046.stdout: "num_objects_misplaced": 0, 2024-09-18T15:56:38.781 INFO:tasks.workunit.client.0.smithi046.stdout: "num_objects_unfound": 0, 2024-09-18T15:56:38.781 INFO:tasks.workunit.client.0.smithi046.stdout: "num_objects_dirty": 30, 2024-09-18T15:56:38.781 INFO:tasks.workunit.client.0.smithi046.stdout: "num_whiteouts": 0, 2024-09-18T15:56:38.781 INFO:tasks.workunit.client.0.smithi046.stdout: "num_read": 0, 2024-09-18T15:56:38.781 INFO:tasks.workunit.client.0.smithi046.stdout: "num_read_kb": 0, 2024-09-18T15:56:38.781 INFO:tasks.workunit.client.0.smithi046.stdout: "num_write": 30, 2024-09-18T15:56:38.781 INFO:tasks.workunit.client.0.smithi046.stdout: "num_write_kb": 30, 2024-09-18T15:56:38.781 INFO:tasks.workunit.client.0.smithi046.stdout: "num_scrub_errors": 0, 2024-09-18T15:56:38.782 INFO:tasks.workunit.client.0.smithi046.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T15:56:38.782 INFO:tasks.workunit.client.0.smithi046.stdout: "num_deep_scrub_errors": 0, 2024-09-18T15:56:38.782 INFO:tasks.workunit.client.0.smithi046.stdout: "num_objects_recovered": 20, 2024-09-18T15:56:38.782 INFO:tasks.workunit.client.0.smithi046.stdout: "num_bytes_recovered": 140, 2024-09-18T15:56:38.782 INFO:tasks.workunit.client.0.smithi046.stdout: "num_keys_recovered": 0, 2024-09-18T15:56:38.782 INFO:tasks.workunit.client.0.smithi046.stdout: "num_objects_omap": 0, 2024-09-18T15:56:38.782 INFO:tasks.workunit.client.0.smithi046.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T15:56:38.782 INFO:tasks.workunit.client.0.smithi046.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T15:56:38.782 INFO:tasks.workunit.client.0.smithi046.stdout: "num_flush": 0, 2024-09-18T15:56:38.782 INFO:tasks.workunit.client.0.smithi046.stdout: "num_flush_kb": 0, 2024-09-18T15:56:38.782 INFO:tasks.workunit.client.0.smithi046.stdout: "num_evict": 0, 2024-09-18T15:56:38.782 INFO:tasks.workunit.client.0.smithi046.stdout: "num_evict_kb": 0, 2024-09-18T15:56:38.782 INFO:tasks.workunit.client.0.smithi046.stdout: "num_promote": 0, 2024-09-18T15:56:38.782 INFO:tasks.workunit.client.0.smithi046.stdout: "num_flush_mode_high": 0, 2024-09-18T15:56:38.782 INFO:tasks.workunit.client.0.smithi046.stdout: "num_flush_mode_low": 0, 2024-09-18T15:56:38.783 INFO:tasks.workunit.client.0.smithi046.stdout: "num_evict_mode_some": 0, 2024-09-18T15:56:38.783 INFO:tasks.workunit.client.0.smithi046.stdout: "num_evict_mode_full": 0, 2024-09-18T15:56:38.783 INFO:tasks.workunit.client.0.smithi046.stdout: "num_objects_pinned": 0, 2024-09-18T15:56:38.783 INFO:tasks.workunit.client.0.smithi046.stdout: "num_legacy_snapsets": 0, 2024-09-18T15:56:38.783 INFO:tasks.workunit.client.0.smithi046.stdout: "num_large_omap_objects": 0, 2024-09-18T15:56:38.783 INFO:tasks.workunit.client.0.smithi046.stdout: "num_objects_manifest": 0, 2024-09-18T15:56:38.783 INFO:tasks.workunit.client.0.smithi046.stdout: "num_omap_bytes": 0, 2024-09-18T15:56:38.783 INFO:tasks.workunit.client.0.smithi046.stdout: "num_omap_keys": 0, 2024-09-18T15:56:38.783 INFO:tasks.workunit.client.0.smithi046.stdout: "num_objects_repaired": 20 2024-09-18T15:56:38.783 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T15:56:38.784 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:708: TEST_repair_stats: ceph pg 1.0 query 2024-09-18T15:56:38.784 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:708: TEST_repair_stats: jq .info.stats.stat_sum.num_objects_repaired 2024-09-18T15:56:38.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:708: TEST_repair_stats: COUNT=20 2024-09-18T15:56:38.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:709: TEST_repair_stats: test 20 = 20 2024-09-18T15:56:38.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2024-09-18T15:56:38.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-09-18T15:56:39.214 INFO:tasks.workunit.client.0.smithi046.stderr:dumped all 2024-09-18T15:56:39.222 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T15:56:39.222 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T15:56:39.222 INFO:tasks.workunit.client.0.smithi046.stdout: "up_from": 20, 2024-09-18T15:56:39.222 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 85899345928, 2024-09-18T15:56:39.223 INFO:tasks.workunit.client.0.smithi046.stdout: "num_pgs": 1, 2024-09-18T15:56:39.223 INFO:tasks.workunit.client.0.smithi046.stdout: "num_osds": 1, 2024-09-18T15:56:39.223 INFO:tasks.workunit.client.0.smithi046.stdout: "num_per_pool_osds": 1, 2024-09-18T15:56:39.223 INFO:tasks.workunit.client.0.smithi046.stdout: "num_per_pool_omap_osds": 1, 2024-09-18T15:56:39.223 INFO:tasks.workunit.client.0.smithi046.stdout: "kb": 104857600, 2024-09-18T15:56:39.223 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used": 298056, 2024-09-18T15:56:39.223 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used_data": 320, 2024-09-18T15:56:39.223 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used_omap": 4, 2024-09-18T15:56:39.223 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used_meta": 297723, 2024-09-18T15:56:39.223 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_avail": 104559544, 2024-09-18T15:56:39.223 INFO:tasks.workunit.client.0.smithi046.stdout: "statfs": { 2024-09-18T15:56:39.223 INFO:tasks.workunit.client.0.smithi046.stdout: "total": 107374182400, 2024-09-18T15:56:39.223 INFO:tasks.workunit.client.0.smithi046.stdout: "available": 107068973056, 2024-09-18T15:56:39.223 INFO:tasks.workunit.client.0.smithi046.stdout: "internally_reserved": 0, 2024-09-18T15:56:39.223 INFO:tasks.workunit.client.0.smithi046.stdout: "allocated": 327680, 2024-09-18T15:56:39.224 INFO:tasks.workunit.client.0.smithi046.stdout: "data_stored": 53374, 2024-09-18T15:56:39.224 INFO:tasks.workunit.client.0.smithi046.stdout: "data_compressed": 0, 2024-09-18T15:56:39.224 INFO:tasks.workunit.client.0.smithi046.stdout: "data_compressed_allocated": 0, 2024-09-18T15:56:39.224 INFO:tasks.workunit.client.0.smithi046.stdout: "data_compressed_original": 0, 2024-09-18T15:56:39.224 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_allocated": 4543, 2024-09-18T15:56:39.224 INFO:tasks.workunit.client.0.smithi046.stdout: "internal_metadata": 304868929 2024-09-18T15:56:39.224 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:56:39.224 INFO:tasks.workunit.client.0.smithi046.stdout: "hb_peers": [ 2024-09-18T15:56:39.224 INFO:tasks.workunit.client.0.smithi046.stdout: 0 2024-09-18T15:56:39.224 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T15:56:39.224 INFO:tasks.workunit.client.0.smithi046.stdout: "snap_trim_queue_len": 0, 2024-09-18T15:56:39.224 INFO:tasks.workunit.client.0.smithi046.stdout: "num_snap_trimming": 0, 2024-09-18T15:56:39.224 INFO:tasks.workunit.client.0.smithi046.stdout: "num_shards_repaired": 10, 2024-09-18T15:56:39.225 INFO:tasks.workunit.client.0.smithi046.stdout: "op_queue_age_hist": { 2024-09-18T15:56:39.225 INFO:tasks.workunit.client.0.smithi046.stdout: "histogram": [], 2024-09-18T15:56:39.225 INFO:tasks.workunit.client.0.smithi046.stdout: "upper_bound": 1 2024-09-18T15:56:39.225 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:56:39.225 INFO:tasks.workunit.client.0.smithi046.stdout: "perf_stat": { 2024-09-18T15:56:39.225 INFO:tasks.workunit.client.0.smithi046.stdout: "commit_latency_ms": 0, 2024-09-18T15:56:39.225 INFO:tasks.workunit.client.0.smithi046.stdout: "apply_latency_ms": 0, 2024-09-18T15:56:39.225 INFO:tasks.workunit.client.0.smithi046.stdout: "commit_latency_ns": 0, 2024-09-18T15:56:39.225 INFO:tasks.workunit.client.0.smithi046.stdout: "apply_latency_ns": 0 2024-09-18T15:56:39.225 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:56:39.225 INFO:tasks.workunit.client.0.smithi046.stdout: "alerts": [], 2024-09-18T15:56:39.225 INFO:tasks.workunit.client.0.smithi046.stdout: "network_ping_times": [ 2024-09-18T15:56:39.226 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T15:56:39.226 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T15:56:39.226 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2024-09-18T15:56:39.226 INFO:tasks.workunit.client.0.smithi046.stdout: "interfaces": [ 2024-09-18T15:56:39.226 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T15:56:39.226 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-09-18T15:56:39.226 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-09-18T15:56:39.226 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0, 2024-09-18T15:56:39.226 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0, 2024-09-18T15:56:39.226 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0 2024-09-18T15:56:39.226 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:56:39.226 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-09-18T15:56:39.226 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0, 2024-09-18T15:56:39.226 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0, 2024-09-18T15:56:39.227 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0 2024-09-18T15:56:39.227 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:56:39.227 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-09-18T15:56:39.227 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0, 2024-09-18T15:56:39.227 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0, 2024-09-18T15:56:39.227 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0 2024-09-18T15:56:39.227 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:56:39.227 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.388 2024-09-18T15:56:39.227 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T15:56:39.227 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T15:56:39.227 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T15:56:39.227 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T15:56:39.227 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T15:56:39.228 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-09-18T15:56:39.228 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2024-09-18T15:56:39.517 INFO:tasks.workunit.client.0.smithi046.stderr:dumped all 2024-09-18T15:56:39.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: COUNT=10 2024-09-18T15:56:39.527 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:713: TEST_repair_stats: expr 20 / 2 2024-09-18T15:56:39.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:713: TEST_repair_stats: test 10 = 10 2024-09-18T15:56:39.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-09-18T15:56:39.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-09-18T15:56:39.820 INFO:tasks.workunit.client.0.smithi046.stderr:dumped all 2024-09-18T15:56:39.828 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T15:56:39.828 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T15:56:39.828 INFO:tasks.workunit.client.0.smithi046.stdout: "up_from": 23, 2024-09-18T15:56:39.828 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 98784247815, 2024-09-18T15:56:39.829 INFO:tasks.workunit.client.0.smithi046.stdout: "num_pgs": 1, 2024-09-18T15:56:39.829 INFO:tasks.workunit.client.0.smithi046.stdout: "num_osds": 1, 2024-09-18T15:56:39.829 INFO:tasks.workunit.client.0.smithi046.stdout: "num_per_pool_osds": 1, 2024-09-18T15:56:39.829 INFO:tasks.workunit.client.0.smithi046.stdout: "num_per_pool_omap_osds": 1, 2024-09-18T15:56:39.829 INFO:tasks.workunit.client.0.smithi046.stdout: "kb": 104857600, 2024-09-18T15:56:39.829 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used": 298312, 2024-09-18T15:56:39.829 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used_data": 320, 2024-09-18T15:56:39.829 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used_omap": 4, 2024-09-18T15:56:39.829 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used_meta": 297979, 2024-09-18T15:56:39.829 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_avail": 104559288, 2024-09-18T15:56:39.829 INFO:tasks.workunit.client.0.smithi046.stdout: "statfs": { 2024-09-18T15:56:39.829 INFO:tasks.workunit.client.0.smithi046.stdout: "total": 107374182400, 2024-09-18T15:56:39.830 INFO:tasks.workunit.client.0.smithi046.stdout: "available": 107068710912, 2024-09-18T15:56:39.830 INFO:tasks.workunit.client.0.smithi046.stdout: "internally_reserved": 0, 2024-09-18T15:56:39.830 INFO:tasks.workunit.client.0.smithi046.stdout: "allocated": 327680, 2024-09-18T15:56:39.830 INFO:tasks.workunit.client.0.smithi046.stdout: "data_stored": 53374, 2024-09-18T15:56:39.830 INFO:tasks.workunit.client.0.smithi046.stdout: "data_compressed": 0, 2024-09-18T15:56:39.830 INFO:tasks.workunit.client.0.smithi046.stdout: "data_compressed_allocated": 0, 2024-09-18T15:56:39.830 INFO:tasks.workunit.client.0.smithi046.stdout: "data_compressed_original": 0, 2024-09-18T15:56:39.830 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_allocated": 4543, 2024-09-18T15:56:39.830 INFO:tasks.workunit.client.0.smithi046.stdout: "internal_metadata": 305131073 2024-09-18T15:56:39.830 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:56:39.830 INFO:tasks.workunit.client.0.smithi046.stdout: "hb_peers": [ 2024-09-18T15:56:39.830 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T15:56:39.830 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T15:56:39.830 INFO:tasks.workunit.client.0.smithi046.stdout: "snap_trim_queue_len": 0, 2024-09-18T15:56:39.831 INFO:tasks.workunit.client.0.smithi046.stdout: "num_snap_trimming": 0, 2024-09-18T15:56:39.831 INFO:tasks.workunit.client.0.smithi046.stdout: "num_shards_repaired": 10, 2024-09-18T15:56:39.831 INFO:tasks.workunit.client.0.smithi046.stdout: "op_queue_age_hist": { 2024-09-18T15:56:39.831 INFO:tasks.workunit.client.0.smithi046.stdout: "histogram": [], 2024-09-18T15:56:39.831 INFO:tasks.workunit.client.0.smithi046.stdout: "upper_bound": 1 2024-09-18T15:56:39.831 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:56:39.831 INFO:tasks.workunit.client.0.smithi046.stdout: "perf_stat": { 2024-09-18T15:56:39.831 INFO:tasks.workunit.client.0.smithi046.stdout: "commit_latency_ms": 0, 2024-09-18T15:56:39.831 INFO:tasks.workunit.client.0.smithi046.stdout: "apply_latency_ms": 0, 2024-09-18T15:56:39.831 INFO:tasks.workunit.client.0.smithi046.stdout: "commit_latency_ns": 0, 2024-09-18T15:56:39.831 INFO:tasks.workunit.client.0.smithi046.stdout: "apply_latency_ns": 0 2024-09-18T15:56:39.831 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:56:39.831 INFO:tasks.workunit.client.0.smithi046.stdout: "alerts": [], 2024-09-18T15:56:39.832 INFO:tasks.workunit.client.0.smithi046.stdout: "network_ping_times": [ 2024-09-18T15:56:39.832 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T15:56:39.832 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T15:56:39.832 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2024-09-18T15:56:39.832 INFO:tasks.workunit.client.0.smithi046.stdout: "interfaces": [ 2024-09-18T15:56:39.832 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T15:56:39.832 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-09-18T15:56:39.832 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-09-18T15:56:39.832 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0, 2024-09-18T15:56:39.832 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0, 2024-09-18T15:56:39.832 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0 2024-09-18T15:56:39.832 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:56:39.832 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-09-18T15:56:39.832 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0, 2024-09-18T15:56:39.833 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0, 2024-09-18T15:56:39.833 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0 2024-09-18T15:56:39.833 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:56:39.833 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-09-18T15:56:39.833 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0, 2024-09-18T15:56:39.833 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0, 2024-09-18T15:56:39.833 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0 2024-09-18T15:56:39.833 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:56:39.833 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.411 2024-09-18T15:56:39.833 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T15:56:39.833 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T15:56:39.833 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T15:56:39.833 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T15:56:39.833 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T15:56:39.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-09-18T15:56:39.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-09-18T15:56:40.117 INFO:tasks.workunit.client.0.smithi046.stderr:dumped all 2024-09-18T15:56:40.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: COUNT=10 2024-09-18T15:56:40.128 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:717: TEST_repair_stats: expr 20 / 2 2024-09-18T15:56:40.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:717: TEST_repair_stats: test 10 = 10 2024-09-18T15:56:40.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-09-18T15:56:40.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: jq .pg_map.osd_stats_sum 2024-09-18T15:56:40.416 INFO:tasks.workunit.client.0.smithi046.stderr:dumped all 2024-09-18T15:56:40.424 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T15:56:40.425 INFO:tasks.workunit.client.0.smithi046.stdout: "up_from": 0, 2024-09-18T15:56:40.425 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 0, 2024-09-18T15:56:40.425 INFO:tasks.workunit.client.0.smithi046.stdout: "num_pgs": 2, 2024-09-18T15:56:40.425 INFO:tasks.workunit.client.0.smithi046.stdout: "num_osds": 2, 2024-09-18T15:56:40.425 INFO:tasks.workunit.client.0.smithi046.stdout: "num_per_pool_osds": 2, 2024-09-18T15:56:40.425 INFO:tasks.workunit.client.0.smithi046.stdout: "num_per_pool_omap_osds": 2, 2024-09-18T15:56:40.425 INFO:tasks.workunit.client.0.smithi046.stdout: "kb": 209715200, 2024-09-18T15:56:40.425 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used": 596368, 2024-09-18T15:56:40.425 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used_data": 640, 2024-09-18T15:56:40.425 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used_omap": 8, 2024-09-18T15:56:40.425 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used_meta": 595703, 2024-09-18T15:56:40.425 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_avail": 209118832, 2024-09-18T15:56:40.425 INFO:tasks.workunit.client.0.smithi046.stdout: "statfs": { 2024-09-18T15:56:40.426 INFO:tasks.workunit.client.0.smithi046.stdout: "total": 214748364800, 2024-09-18T15:56:40.426 INFO:tasks.workunit.client.0.smithi046.stdout: "available": 214137683968, 2024-09-18T15:56:40.426 INFO:tasks.workunit.client.0.smithi046.stdout: "internally_reserved": 0, 2024-09-18T15:56:40.426 INFO:tasks.workunit.client.0.smithi046.stdout: "allocated": 655360, 2024-09-18T15:56:40.426 INFO:tasks.workunit.client.0.smithi046.stdout: "data_stored": 106748, 2024-09-18T15:56:40.426 INFO:tasks.workunit.client.0.smithi046.stdout: "data_compressed": 0, 2024-09-18T15:56:40.426 INFO:tasks.workunit.client.0.smithi046.stdout: "data_compressed_allocated": 0, 2024-09-18T15:56:40.426 INFO:tasks.workunit.client.0.smithi046.stdout: "data_compressed_original": 0, 2024-09-18T15:56:40.426 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_allocated": 9086, 2024-09-18T15:56:40.426 INFO:tasks.workunit.client.0.smithi046.stdout: "internal_metadata": 610000002 2024-09-18T15:56:40.426 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:56:40.426 INFO:tasks.workunit.client.0.smithi046.stdout: "hb_peers": [], 2024-09-18T15:56:40.426 INFO:tasks.workunit.client.0.smithi046.stdout: "snap_trim_queue_len": 0, 2024-09-18T15:56:40.426 INFO:tasks.workunit.client.0.smithi046.stdout: "num_snap_trimming": 0, 2024-09-18T15:56:40.427 INFO:tasks.workunit.client.0.smithi046.stdout: "num_shards_repaired": 20, 2024-09-18T15:56:40.427 INFO:tasks.workunit.client.0.smithi046.stdout: "op_queue_age_hist": { 2024-09-18T15:56:40.427 INFO:tasks.workunit.client.0.smithi046.stdout: "histogram": [], 2024-09-18T15:56:40.427 INFO:tasks.workunit.client.0.smithi046.stdout: "upper_bound": 1 2024-09-18T15:56:40.427 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:56:40.427 INFO:tasks.workunit.client.0.smithi046.stdout: "perf_stat": { 2024-09-18T15:56:40.427 INFO:tasks.workunit.client.0.smithi046.stdout: "commit_latency_ms": 0, 2024-09-18T15:56:40.427 INFO:tasks.workunit.client.0.smithi046.stdout: "apply_latency_ms": 0, 2024-09-18T15:56:40.427 INFO:tasks.workunit.client.0.smithi046.stdout: "commit_latency_ns": 0, 2024-09-18T15:56:40.427 INFO:tasks.workunit.client.0.smithi046.stdout: "apply_latency_ns": 0 2024-09-18T15:56:40.427 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:56:40.427 INFO:tasks.workunit.client.0.smithi046.stdout: "alerts": [], 2024-09-18T15:56:40.427 INFO:tasks.workunit.client.0.smithi046.stdout: "network_ping_times": [] 2024-09-18T15:56:40.427 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T15:56:40.428 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:720: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-09-18T15:56:40.428 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:720: TEST_repair_stats: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-09-18T15:56:40.718 INFO:tasks.workunit.client.0.smithi046.stderr:dumped all 2024-09-18T15:56:40.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:720: TEST_repair_stats: COUNT=20 2024-09-18T15:56:40.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:721: TEST_repair_stats: test 20 = 20 2024-09-18T15:56:40.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T15:56:40.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:56:40.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:56:40.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:56:40.729 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:56:40.729 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:56:40.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:56:40.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:56:40.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:56:40.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:56:40.847 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:56:40.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:56:40.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:56:40.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:56:40.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:56:40.853 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:56:40.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:56:40.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:56:40.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:56:40.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:56:40.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:56:40.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:56:40.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:56:40.879 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:56:40.879 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:56:40.879 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:56:40.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:56:40.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:56:40.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:56:40.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T15:56:40.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T15:56:40.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T15:56:40.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T15:56:40.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:56:40.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:56:40.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:56:40.882 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:56:40.882 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:56:40.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:56:40.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:56:40.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:56:40.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:56:40.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:56:40.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:56:40.891 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:56:40.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:56:40.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:56:40.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:56:40.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:56:40.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:56:40.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:56:40.897 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:56:40.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:56:40.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:56:40.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:56:40.904 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:56:40.904 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:56:40.904 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:56:40.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:56:40.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:56:40.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:56:40.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T15:56:40.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:56:40.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:56:40.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:56:40.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T15:56:40.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:56:40.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:56:40.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:56:40.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T15:56:40.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_repair_stats_ec td/osd-scrub-repair 2024-09-18T15:56:40.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats_ec: local dir=td/osd-scrub-repair 2024-09-18T15:56:40.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:726: TEST_repair_stats_ec: local poolname=testpool 2024-09-18T15:56:40.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:727: TEST_repair_stats_ec: local OSDS=3 2024-09-18T15:56:40.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:728: TEST_repair_stats_ec: local OBJS=30 2024-09-18T15:56:40.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:730: TEST_repair_stats_ec: local REPAIRS=26 2024-09-18T15:56:40.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:731: TEST_repair_stats_ec: local allow_overwrites=false 2024-09-18T15:56:40.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:734: TEST_repair_stats_ec: run_mon td/osd-scrub-repair a 2024-09-18T15:56:40.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T15:56:40.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:56:40.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:56:40.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:56:40.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T15:56:40.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-18T15:56:41.295 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:56:41.295 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:56:41.295 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:56:41.295 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:56:41.295 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:56:41.295 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:56:41.296 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:56:41.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T15:56:41.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:56:41.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:56:41.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:56:41.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:56:41.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:56:41.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:56:41.569 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:56:41.569 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:56:41.569 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:56:41.570 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:56:41.570 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:56:41.571 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:56:41.571 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:56:41.572 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:56:41.572 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T15:56:41.669 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:56:41.669 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:56:41.669 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:56:41.669 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:56:41.669 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:56:41.670 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:56:41.670 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:56:41.670 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:56:41.671 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:56:41.671 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:56:41.671 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:56:41.672 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:56:41.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:56:41.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T15:56:41.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:735: TEST_repair_stats_ec: run_mgr td/osd-scrub-repair x 2024-09-18T15:56:41.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T15:56:41.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:56:41.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:56:41.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:56:41.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T15:56:41.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:56:42.629 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:56:42.629 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:56:42.629 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:56:42.630 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:56:42.630 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:56:42.630 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:56:42.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:56:42.632 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:56:42.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:56:42.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:56:42.670 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: expr 3 - 1 2024-09-18T15:56:42.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: seq 0 2 2024-09-18T15:56:42.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T15:56:42.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:739: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:56:42.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:56:42.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:56:42.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:56:42.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:56:42.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:56:42.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:56:42.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:56:42.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:56:42.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:56:42.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:56:42.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:56:42.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:56:42.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:56:42.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:56:42.676 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:56:42.677 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:56:42.677 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:56:42.678 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:56:42.679 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:56:42.679 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:56:42.679 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:56:42.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:56:42.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:56:42.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:56:42.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:56:42.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:56:42.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:56:42.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:56:42.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:56:42.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:56:42.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:56:42.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:56:42.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:56:42.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:56:42.680 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:56:42.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=afb8fc3c-6e91-47f8-a6ce-e33ad95d61fc 2024-09-18T15:56:42.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 afb8fc3c-6e91-47f8-a6ce-e33ad95d61fc' 2024-09-18T15:56:42.681 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 afb8fc3c-6e91-47f8-a6ce-e33ad95d61fc 2024-09-18T15:56:42.682 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:56:42.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA6+Opmlxy6KRAAT5DI1Cv2znXL40+9DDr82A== 2024-09-18T15:56:42.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA6+Opmlxy6KRAAT5DI1Cv2znXL40+9DDr82A=="}' 2024-09-18T15:56:42.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new afb8fc3c-6e91-47f8-a6ce-e33ad95d61fc -i td/osd-scrub-repair/0/new.json 2024-09-18T15:56:42.971 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:56:42.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-18T15:56:42.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQA6+Opmlxy6KRAAT5DI1Cv2znXL40+9DDr82A== --osd-uuid afb8fc3c-6e91-47f8-a6ce-e33ad95d61fc 2024-09-18T15:56:43.013 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:56:43.006+0000 7fa2bf2bf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:56:43.017 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:56:43.010+0000 7fa2bf2bf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:56:43.019 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:56:43.014+0000 7fa2bf2bf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:56:43.020 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:56:43.014+0000 7fa2bf2bf540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:56:43.020 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:56:43.014+0000 7fa2bf2bf540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:56:43.020 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:56:43.014+0000 7fa2bf2bf540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T15:56:43.103 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:56:43.098+0000 7fa2bf2bf540 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-18T15:56:48.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T15:56:48.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:56:48.638 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T15:56:48.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:56:48.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:56:49.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:56:49.019 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T15:56:49.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:56:49.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:56:49.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:56:49.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:56:49.054 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:56:49.046+0000 7fe640b02540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:56:49.057 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:56:49.050+0000 7fe640b02540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:56:49.059 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:56:49.054+0000 7fe640b02540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:56:49.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:56:49.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:56:49.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:56:49.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:56:49.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:56:49.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:49.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:56:49.306 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:56:49.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:49.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:49.605 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:56:49.598+0000 7fe640b02540 -1 Falling back to public interface 2024-09-18T15:56:49.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:50.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:50.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:50.660 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:56:50.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:56:50.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:50.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:50.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:51.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:51.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:51.963 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:56:51.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:56:51.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:51.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:52.233 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:56:52.226+0000 7fe640b02540 -1 osd.0 0 log_to_monitors true 2024-09-18T15:56:52.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:53.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:53.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:53.283 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:56:53.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:56:53.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:53.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:53.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:54.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:54.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:54.591 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:56:54.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:56:54.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:54.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:54.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:55.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:55.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:55.910 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:56:55.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:56:55.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:55.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:56.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:57.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:57.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:57.228 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:56:57.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:56:57.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:57.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:57.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:58.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:58.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:58.541 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:56:58.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:56:58.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:58.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:58.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:59.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:59.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:59.953 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:56:59.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:56:59.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:59.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:57:00.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:01.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:01.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:01.273 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:57:01.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:57:01.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:01.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:57:01.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:02.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:02.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:02.591 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:57:02.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:57:02.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:02.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:57:02.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:03.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:03.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:03.914 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:57:03.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:57:03.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:03.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:57:04.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:05.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:05.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:05.559 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:57:05.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:57:05.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:05.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:57:05.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:06.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:06.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:06.985 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:57:06.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:57:06.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:06.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:57:07.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:08.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:08.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:08.705 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:57:08.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:57:08.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:08.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:57:09.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:10.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:10.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:10.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:57:10.415 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:57:10.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:10.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:57:11.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:12.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:12.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:12.020 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:57:12.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:57:12.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:12.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:57:12.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:13.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:13.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:13.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:57:13.590 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:57:13.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:13.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:57:14.037 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/445254,v1:127.0.0.1:6803/445254] [v2:127.0.0.1:6804/445254,v1:127.0.0.1:6805/445254] exists,up afb8fc3c-6e91-47f8-a6ce-e33ad95d61fc 2024-09-18T15:57:14.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:57:14.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:57:14.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:57:14.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T15:57:14.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:739: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:57:14.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:57:14.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:57:14.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:57:14.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:57:14.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:57:14.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:57:14.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:57:14.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:57:14.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:57:14.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:57:14.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:57:14.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:57:14.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:57:14.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:57:14.041 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:57:14.041 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:57:14.041 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:57:14.041 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:57:14.041 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:57:14.041 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:57:14.041 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:57:14.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:57:14.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:57:14.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:57:14.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:57:14.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:57:14.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:57:14.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:57:14.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:57:14.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:57:14.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:57:14.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:57:14.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:57:14.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:57:14.045 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:57:14.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cbd0a5ba-dbbe-4416-8718-cef63e6222a8 2024-09-18T15:57:14.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 cbd0a5ba-dbbe-4416-8718-cef63e6222a8' 2024-09-18T15:57:14.047 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 cbd0a5ba-dbbe-4416-8718-cef63e6222a8 2024-09-18T15:57:14.048 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:57:14.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBa+OpmIaoABBAAK8CxHrmgXlxBi0SXF3I6sQ== 2024-09-18T15:57:14.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBa+OpmIaoABBAAK8CxHrmgXlxBi0SXF3I6sQ=="}' 2024-09-18T15:57:14.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cbd0a5ba-dbbe-4416-8718-cef63e6222a8 -i td/osd-scrub-repair/1/new.json 2024-09-18T15:57:14.573 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:57:14.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-18T15:57:14.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBa+OpmIaoABBAAK8CxHrmgXlxBi0SXF3I6sQ== --osd-uuid cbd0a5ba-dbbe-4416-8718-cef63e6222a8 2024-09-18T15:57:14.629 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:57:14.621+0000 7f3444d7b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:57:14.633 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:57:14.625+0000 7f3444d7b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:57:14.635 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:57:14.629+0000 7f3444d7b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:57:14.636 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:57:14.629+0000 7f3444d7b540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:57:14.636 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:57:14.629+0000 7f3444d7b540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:57:14.636 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:57:14.629+0000 7f3444d7b540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T15:57:14.705 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:57:14.697+0000 7f3444d7b540 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-18T15:57:20.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-18T15:57:20.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:57:20.719 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T15:57:20.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:57:20.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:57:21.110 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T15:57:21.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:57:21.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:57:21.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:57:21.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:57:21.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:57:21.143 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:57:21.137+0000 7f62e32c5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:57:21.145 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:57:21.137+0000 7f62e32c5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:57:21.146 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:57:21.137+0000 7f62e32c5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:57:21.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:57:21.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:57:21.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:57:21.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:57:21.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:57:21.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:21.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:57:21.459 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:57:21.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:21.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:21.684 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:57:21.677+0000 7f62e32c5540 -1 Falling back to public interface 2024-09-18T15:57:21.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:22.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:22.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:22.798 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:57:22.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:57:22.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:22.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:23.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:24.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:24.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:24.114 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:57:24.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:57:24.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:24.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:24.218 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:57:24.209+0000 7f62e32c5540 -1 osd.1 0 log_to_monitors true 2024-09-18T15:57:24.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:25.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:25.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:25.496 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:57:25.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:57:25.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:25.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:25.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:26.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:26.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:26.822 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:57:26.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:57:26.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:26.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:27.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:28.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:28.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:28.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:57:28.311 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:57:28.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:28.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:28.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:29.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:29.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:29.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:57:29.633 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:57:29.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:29.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:30.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:31.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:31.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:31.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:57:31.113 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:57:31.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:31.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:31.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:32.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:32.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:32.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:57:32.541 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:57:32.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:32.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:32.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:33.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:33.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:33.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:57:33.857 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:57:33.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:33.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:34.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:35.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:35.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:35.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:57:35.289 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:57:35.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:35.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:35.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:36.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:36.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:36.609 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:57:36.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:57:36.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:36.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:36.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:37.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:37.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:37.928 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:57:37.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:57:37.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:37.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:38.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:39.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:39.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:39.785 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:57:39.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:57:39.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:39.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:40.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:41.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:41.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:41.241 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:57:41.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:57:41.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:41.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:41.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:42.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:42.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:42.968 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:57:42.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:57:42.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:42.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:43.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:44.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:44.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:44.593 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:57:44.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:57:44.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:44.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:45.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:46.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:46.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:46.075 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:57:46.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:57:46.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:46.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:46.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:47.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:47.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:47.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:57:47.748 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:57:47.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:47.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:48.073 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/447008,v1:127.0.0.1:6811/447008] [v2:127.0.0.1:6812/447008,v1:127.0.0.1:6813/447008] exists,up cbd0a5ba-dbbe-4416-8718-cef63e6222a8 2024-09-18T15:57:48.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:57:48.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:57:48.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:57:48.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T15:57:48.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:739: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 2 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:57:48.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:57:48.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:57:48.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T15:57:48.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:57:48.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-18T15:57:48.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:57:48.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:57:48.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:57:48.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:57:48.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T15:57:48.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T15:57:48.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:57:48.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:57:48.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:57:48.078 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:57:48.078 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:57:48.078 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:57:48.078 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:57:48.078 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:57:48.078 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:57:48.078 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:57:48.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:57:48.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:57:48.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:57:48.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:57:48.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:57:48.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:57:48.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:57:48.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:57:48.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:57:48.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:57:48.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:57:48.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:57:48.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-18T15:57:48.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:57:48.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cc80bcce-4d8b-4b22-8a56-3ad7873d07d0 2024-09-18T15:57:48.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 cc80bcce-4d8b-4b22-8a56-3ad7873d07d0' 2024-09-18T15:57:48.083 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 cc80bcce-4d8b-4b22-8a56-3ad7873d07d0 2024-09-18T15:57:48.084 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:57:48.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB8+OpmE8YQBhAAwk8FNmQrP5k8L0CHqiz39A== 2024-09-18T15:57:48.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB8+OpmE8YQBhAAwk8FNmQrP5k8L0CHqiz39A=="}' 2024-09-18T15:57:48.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cc80bcce-4d8b-4b22-8a56-3ad7873d07d0 -i td/osd-scrub-repair/2/new.json 2024-09-18T15:57:48.530 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:57:48.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-18T15:57:48.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQB8+OpmE8YQBhAAwk8FNmQrP5k8L0CHqiz39A== --osd-uuid cc80bcce-4d8b-4b22-8a56-3ad7873d07d0 2024-09-18T15:57:48.585 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:57:48.577+0000 7fa665479540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:57:48.588 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:57:48.581+0000 7fa665479540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:57:48.590 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:57:48.581+0000 7fa665479540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:57:48.590 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:57:48.581+0000 7fa665479540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T15:57:48.591 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:57:48.581+0000 7fa665479540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T15:57:48.591 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:57:48.581+0000 7fa665479540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T15:57:48.649 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:57:48.641+0000 7fa665479540 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-18T15:57:54.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-18T15:57:54.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:57:54.970 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T15:57:54.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T15:57:54.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:57:55.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T15:57:55.361 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T15:57:55.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:57:55.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:57:55.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:57:55.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:57:55.395 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:57:55.389+0000 7f707caa6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:57:55.399 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:57:55.389+0000 7f707caa6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:57:55.400 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:57:55.393+0000 7f707caa6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:57:55.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T15:57:55.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:57:55.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:57:55.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:57:55.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:57:55.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:55.653 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:57:55.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:57:55.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:55.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:57:55.957 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:57:55.949+0000 7f707caa6540 -1 Falling back to public interface 2024-09-18T15:57:55.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:56.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:56.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:56.990 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:57:56.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:57:56.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:56.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:57:57.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:58.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:58.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:58.306 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:57:58.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:57:58.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:58.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:57:58.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:58.798 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:57:58.789+0000 7f707caa6540 -1 osd.2 0 log_to_monitors true 2024-09-18T15:57:59.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:59.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:59.632 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:57:59.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:57:59.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:59.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:58:00.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:01.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:01.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:01.047 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T15:58:01.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:58:01.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:01.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:58:01.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:02.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:02.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:02.365 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T15:58:02.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:58:02.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:02.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:58:02.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:03.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:03.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:03.678 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T15:58:03.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:58:03.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:03.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:58:03.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:04.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:04.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:04.992 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T15:58:04.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:58:04.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:04.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:58:05.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:06.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:06.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:06.312 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T15:58:06.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:58:06.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:06.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:58:06.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:07.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:07.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:07.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:58:07.644 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T15:58:07.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:07.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:58:07.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:08.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:08.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:08.958 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T15:58:08.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:58:08.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:58:08.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:09.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:10.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:10.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:10.269 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T15:58:10.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:58:10.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:10.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:58:10.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:11.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:11.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:11.649 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T15:58:11.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:58:11.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:11.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:58:12.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:13.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:13.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:13.531 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T15:58:13.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:58:13.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:13.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:58:14.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:15.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:15.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:15.075 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T15:58:15.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:58:15.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:15.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:58:15.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:16.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:16.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:16.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:58:16.426 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T15:58:16.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:16.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:58:17.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:18.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:18.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:18.073 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T15:58:18.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:58:18.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:18.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:58:18.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:19.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:19.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:19.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:58:19.563 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T15:58:19.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:19.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:58:19.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:20.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:20.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:20.955 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T15:58:20.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:58:20.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:20.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:58:21.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:22.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:22.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:22.324 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T15:58:22.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T15:58:22.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:22.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:58:22.638 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/448789,v1:127.0.0.1:6819/448789] [v2:127.0.0.1:6820/448789,v1:127.0.0.1:6821/448789] exists,up cc80bcce-4d8b-4b22-8a56-3ad7873d07d0 2024-09-18T15:58:22.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:58:22.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:58:22.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:58:22.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:743: TEST_repair_stats_ec: create_ec_pool testpool false k=2 m=1 2024-09-18T15:58:22.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: create_ec_pool: local pool_name=testpool 2024-09-18T15:58:22.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: create_ec_pool: shift 2024-09-18T15:58:22.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: create_ec_pool: local allow_overwrites=false 2024-09-18T15:58:22.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: create_ec_pool: shift 2024-09-18T15:58:22.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-18T15:58:23.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: create_ec_pool: create_pool testpool 1 1 erasure myprofile 2024-09-18T15:58:23.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 erasure myprofile 2024-09-18T15:58:24.043 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'testpool' created 2024-09-18T15:58:24.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:58:25.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: create_ec_pool: '[' false = true ']' 2024-09-18T15:58:25.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: create_ec_pool: wait_for_clean 2024-09-18T15:58:25.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:58:25.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:58:25.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:58:25.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:58:25.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:58:25.060 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:58:25.061 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:58:25.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:58:25.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:58:25.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:58:25.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:58:25.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:58:25.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:58:25.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:58:25.295 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:58:25.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:58:25.610 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:58:25.611 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:58:25.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:58:25.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:58:25.611 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:58:25.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836498 2024-09-18T15:58:25.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836498 2024-09-18T15:58:25.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498' 2024-09-18T15:58:25.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:58:25.809 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:58:25.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672974 2024-09-18T15:58:25.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672974 2024-09-18T15:58:25.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498 1-42949672974' 2024-09-18T15:58:25.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:58:25.949 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:58:26.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509449 2024-09-18T15:58:26.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509449 2024-09-18T15:58:26.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498 1-42949672974 2-64424509449' 2024-09-18T15:58:26.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:58:26.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836498 2024-09-18T15:58:26.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:58:26.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:58:26.109 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836498 2024-09-18T15:58:26.109 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:58:26.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836498 2024-09-18T15:58:26.112 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836498 2024-09-18T15:58:26.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-09-18T15:58:26.113 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:58:26.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836498 -lt 21474836498 2024-09-18T15:58:26.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:58:26.436 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672974 2024-09-18T15:58:26.436 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:58:26.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:58:26.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:58:26.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672974 2024-09-18T15:58:26.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672974 2024-09-18T15:58:26.442 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672974 2024-09-18T15:58:26.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-09-18T15:58:26.443 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:58:26.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672974 -lt 42949672974 2024-09-18T15:58:26.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:58:26.787 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509449 2024-09-18T15:58:26.787 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:58:26.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:58:26.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509449 2024-09-18T15:58:26.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:58:26.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509449 2024-09-18T15:58:26.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-18T15:58:26.794 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509449 2024-09-18T15:58:26.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:58:27.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509448 -lt 64424509449 2024-09-18T15:58:27.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:58:28.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:58:28.133 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:58:28.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509450 -lt 64424509449 2024-09-18T15:58:28.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:58:28.452 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:58:28.452 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:58:28.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:58:28.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:58:28.868 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:58:28.868 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:58:28.868 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:58:28.868 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:58:28.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:58:28.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:58:29.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:58:29.204 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:58:29.204 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:58:29.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:58:29.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:58:29.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:58:29.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:58:29.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: create_ec_pool: return 0 2024-09-18T15:58:29.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: local payload=ABCDEF 2024-09-18T15:58:29.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: echo ABCDEF 2024-09-18T15:58:29.550 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: seq 1 30 2024-09-18T15:58:29.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:29.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:29.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:29.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:29.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:29.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:30.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:30.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:30.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:30.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:30.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:30.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:30.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:30.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:31.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:31.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:31.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:31.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:31.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:31.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:31.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:31.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:32.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:32.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:32.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:32.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:32.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:32.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:32.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:32.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:32.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:32.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:33.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:33.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:33.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:33.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:33.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:33.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:33.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:33.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:34.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:34.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:34.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:34.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:34.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:34.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:34.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:34.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:35.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:35.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:35.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:35.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:35.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:35.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:35.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:35.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:35.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:35.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:36.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-18T15:58:36.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:750: TEST_repair_stats_ec: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-09-18T15:58:36.376 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: get_not_primary testpool obj1 2024-09-18T15:58:36.376 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-09-18T15:58:36.376 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-09-18T15:58:36.377 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-09-18T15:58:36.377 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-09-18T15:58:36.377 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:58:36.377 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-09-18T15:58:36.377 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:58:36.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T15:58:36.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-09-18T15:58:36.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T15:58:37.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: local other=0 2024-09-18T15:58:37.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: get_pg testpool obj1 2024-09-18T15:58:37.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-09-18T15:58:37.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-09-18T15:58:37.032 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-09-18T15:58:37.032 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:58:37.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: local pgid=1.0 2024-09-18T15:58:37.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: get_primary testpool obj1 2024-09-18T15:58:37.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-09-18T15:58:37.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:58:37.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-09-18T15:58:37.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:58:37.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats_ec: local primary=1 2024-09-18T15:58:37.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:759: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T15:58:37.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:58:37.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:58:37.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:58:37.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:58:37.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:58:38.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:58:38.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-18T15:58:38.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:58:38.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:58:38.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:58:38.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:58:38.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:58:39.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:58:39.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: seq 1 26 2024-09-18T15:58:39.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-18T15:58:39.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: expr 1 % 2 2024-09-18T15:58:39.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: OSD=1 2024-09-18T15:58:39.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-09-18T15:58:39.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:58:39.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:58:39.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:58:39.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:58:39.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:58:39.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-09-18T15:58:40.845 INFO:tasks.workunit.client.0.smithi046.stdout:remove 0#1:ff7b1f36:::obj1:head# 2024-09-18T15:58:41.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-18T15:58:41.390 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: expr 2 % 2 2024-09-18T15:58:41.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: OSD=0 2024-09-18T15:58:41.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-09-18T15:58:41.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:58:41.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:58:41.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:58:41.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:58:41.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:58:41.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-09-18T15:58:42.880 INFO:tasks.workunit.client.0.smithi046.stdout:remove 1#1:104778fc:::obj2:head# 2024-09-18T15:58:43.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-18T15:58:43.422 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: expr 3 % 2 2024-09-18T15:58:43.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: OSD=1 2024-09-18T15:58:43.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-09-18T15:58:43.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:58:43.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:58:43.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:58:43.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:58:43.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:58:43.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-09-18T15:58:44.474 INFO:tasks.workunit.client.0.smithi046.stdout:remove 0#1:8dd16f86:::obj3:head# 2024-09-18T15:58:45.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-18T15:58:45.258 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: expr 4 % 2 2024-09-18T15:58:45.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: OSD=0 2024-09-18T15:58:45.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-09-18T15:58:45.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:58:45.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:58:45.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:58:45.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:58:45.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:58:45.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-09-18T15:58:46.274 INFO:tasks.workunit.client.0.smithi046.stdout:remove 1#1:0ee9ae15:::obj4:head# 2024-09-18T15:58:46.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-18T15:58:46.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: expr 5 % 2 2024-09-18T15:58:46.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: OSD=1 2024-09-18T15:58:46.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-09-18T15:58:46.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:58:46.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:58:46.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:58:46.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:58:46.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:58:46.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-09-18T15:58:47.723 INFO:tasks.workunit.client.0.smithi046.stdout:remove 0#1:c52c9666:::obj5:head# 2024-09-18T15:58:48.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-18T15:58:48.265 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: expr 6 % 2 2024-09-18T15:58:48.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: OSD=0 2024-09-18T15:58:48.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-09-18T15:58:48.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:58:48.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:58:48.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:58:48.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:58:48.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:58:48.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-09-18T15:58:49.222 INFO:tasks.workunit.client.0.smithi046.stdout:remove 1#1:233a42c1:::obj6:head# 2024-09-18T15:58:49.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-18T15:58:49.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: expr 7 % 2 2024-09-18T15:58:49.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: OSD=1 2024-09-18T15:58:49.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-09-18T15:58:49.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:58:49.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:58:49.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:58:49.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:58:49.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:58:49.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-09-18T15:58:50.673 INFO:tasks.workunit.client.0.smithi046.stdout:remove 0#1:9a09113a:::obj7:head# 2024-09-18T15:58:51.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-18T15:58:51.210 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: expr 8 % 2 2024-09-18T15:58:51.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: OSD=0 2024-09-18T15:58:51.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-09-18T15:58:51.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:58:51.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:58:51.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:58:51.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:58:51.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:58:51.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-09-18T15:58:52.179 INFO:tasks.workunit.client.0.smithi046.stdout:remove 1#1:a557efb1:::obj8:head# 2024-09-18T15:58:52.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-18T15:58:52.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: expr 9 % 2 2024-09-18T15:58:52.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: OSD=1 2024-09-18T15:58:52.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-09-18T15:58:52.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:58:52.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:58:52.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:58:52.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:58:52.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:58:52.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-09-18T15:58:53.641 INFO:tasks.workunit.client.0.smithi046.stdout:remove 0#1:fe351e27:::obj9:head# 2024-09-18T15:58:54.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-18T15:58:54.177 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: expr 10 % 2 2024-09-18T15:58:54.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: OSD=0 2024-09-18T15:58:54.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-09-18T15:58:54.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:58:54.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:58:54.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:58:54.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:58:54.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:58:54.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-09-18T15:58:55.145 INFO:tasks.workunit.client.0.smithi046.stdout:remove 1#1:d1337354:::obj10:head# 2024-09-18T15:58:55.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-18T15:58:55.682 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: expr 11 % 2 2024-09-18T15:58:55.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: OSD=1 2024-09-18T15:58:55.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-09-18T15:58:55.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:58:55.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:58:55.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:58:55.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:58:55.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:58:55.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-09-18T15:58:56.595 INFO:tasks.workunit.client.0.smithi046.stdout:remove 0#1:eeae2a94:::obj11:head# 2024-09-18T15:58:57.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-18T15:58:57.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: expr 12 % 2 2024-09-18T15:58:57.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: OSD=0 2024-09-18T15:58:57.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-09-18T15:58:57.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:58:57.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:58:57.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:58:57.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:58:57.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:58:57.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-09-18T15:58:58.316 INFO:tasks.workunit.client.0.smithi046.stdout:remove 1#1:af5f95cb:::obj12:head# 2024-09-18T15:58:59.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-18T15:58:59.102 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: expr 13 % 2 2024-09-18T15:58:59.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: OSD=1 2024-09-18T15:58:59.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-09-18T15:58:59.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:58:59.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:58:59.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:58:59.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:58:59.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:58:59.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-09-18T15:58:59.985 INFO:tasks.workunit.client.0.smithi046.stdout:remove 0#1:a61ad63e:::obj13:head# 2024-09-18T15:59:00.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-18T15:59:00.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: expr 14 % 2 2024-09-18T15:59:00.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: OSD=0 2024-09-18T15:59:00.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-09-18T15:59:00.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:59:00.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:59:00.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:59:00.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:59:00.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:59:00.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-09-18T15:59:01.727 INFO:tasks.workunit.client.0.smithi046.stdout:remove 1#1:e0a44829:::obj14:head# 2024-09-18T15:59:02.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-18T15:59:02.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: expr 15 % 2 2024-09-18T15:59:02.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: OSD=1 2024-09-18T15:59:02.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-09-18T15:59:02.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:59:02.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:59:02.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:59:02.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:59:02.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:59:02.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-09-18T15:59:03.393 INFO:tasks.workunit.client.0.smithi046.stdout:remove 0#1:ab946124:::obj15:head# 2024-09-18T15:59:03.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-18T15:59:03.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: expr 16 % 2 2024-09-18T15:59:03.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: OSD=0 2024-09-18T15:59:03.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-09-18T15:59:03.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:59:03.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:59:03.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:59:03.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:59:03.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:59:03.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-09-18T15:59:04.820 INFO:tasks.workunit.client.0.smithi046.stdout:remove 1#1:461f8b5e:::obj16:head# 2024-09-18T15:59:05.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-18T15:59:05.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: expr 17 % 2 2024-09-18T15:59:05.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: OSD=1 2024-09-18T15:59:05.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-09-18T15:59:05.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:59:05.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:59:05.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:59:05.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:59:05.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:59:05.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-09-18T15:59:06.266 INFO:tasks.workunit.client.0.smithi046.stdout:remove 0#1:24fc6e92:::obj17:head# 2024-09-18T15:59:06.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-18T15:59:06.801 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: expr 18 % 2 2024-09-18T15:59:06.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: OSD=0 2024-09-18T15:59:06.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-09-18T15:59:06.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:59:06.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:59:06.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:59:06.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:59:06.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:59:06.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-09-18T15:59:07.659 INFO:tasks.workunit.client.0.smithi046.stdout:remove 1#1:b9836a99:::obj18:head# 2024-09-18T15:59:08.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-18T15:59:08.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: expr 19 % 2 2024-09-18T15:59:08.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: OSD=1 2024-09-18T15:59:08.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-09-18T15:59:08.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:59:08.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:59:08.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:59:08.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:59:08.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:59:08.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-09-18T15:59:09.147 INFO:tasks.workunit.client.0.smithi046.stdout:remove 0#1:00b31b6c:::obj19:head# 2024-09-18T15:59:09.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-18T15:59:09.934 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: expr 20 % 2 2024-09-18T15:59:09.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: OSD=0 2024-09-18T15:59:09.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-09-18T15:59:09.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:59:09.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:59:09.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:59:09.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:59:09.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:59:09.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-09-18T15:59:10.834 INFO:tasks.workunit.client.0.smithi046.stdout:remove 1#1:24b5611b:::obj20:head# 2024-09-18T15:59:11.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-18T15:59:11.622 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: expr 21 % 2 2024-09-18T15:59:11.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: OSD=1 2024-09-18T15:59:11.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj21 remove 2024-09-18T15:59:11.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:59:11.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:59:11.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:59:11.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:59:11.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:59:11.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj21 remove 2024-09-18T15:59:12.648 INFO:tasks.workunit.client.0.smithi046.stdout:remove 0#1:1243fe1a:::obj21:head# 2024-09-18T15:59:13.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-18T15:59:13.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: expr 22 % 2 2024-09-18T15:59:13.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: OSD=0 2024-09-18T15:59:13.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj22 remove 2024-09-18T15:59:13.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:59:13.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:59:13.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:59:13.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:59:13.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:59:13.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj22 remove 2024-09-18T15:59:14.341 INFO:tasks.workunit.client.0.smithi046.stdout:remove 1#1:c0c49def:::obj22:head# 2024-09-18T15:59:14.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-18T15:59:14.886 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: expr 23 % 2 2024-09-18T15:59:14.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: OSD=1 2024-09-18T15:59:14.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj23 remove 2024-09-18T15:59:14.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:59:14.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:59:14.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:59:14.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:59:14.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:59:14.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj23 remove 2024-09-18T15:59:15.712 INFO:tasks.workunit.client.0.smithi046.stdout:remove 0#1:242c40e0:::obj23:head# 2024-09-18T15:59:16.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-18T15:59:16.245 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: expr 24 % 2 2024-09-18T15:59:16.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: OSD=0 2024-09-18T15:59:16.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj24 remove 2024-09-18T15:59:16.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:59:16.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:59:16.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:59:16.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:59:16.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:59:16.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj24 remove 2024-09-18T15:59:17.217 INFO:tasks.workunit.client.0.smithi046.stdout:remove 1#1:7eacede5:::obj24:head# 2024-09-18T15:59:17.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-18T15:59:17.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: expr 25 % 2 2024-09-18T15:59:17.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: OSD=1 2024-09-18T15:59:17.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj25 remove 2024-09-18T15:59:17.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:59:17.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:59:17.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:59:17.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:59:17.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:59:17.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj25 remove 2024-09-18T15:59:18.603 INFO:tasks.workunit.client.0.smithi046.stdout:remove 0#1:7109478f:::obj25:head# 2024-09-18T15:59:19.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-18T15:59:19.137 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: expr 26 % 2 2024-09-18T15:59:19.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: OSD=0 2024-09-18T15:59:19.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj26 remove 2024-09-18T15:59:19.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T15:59:19.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:59:19.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T15:59:19.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:59:19.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:59:19.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj26 remove 2024-09-18T15:59:20.023 INFO:tasks.workunit.client.0.smithi046.stdout:remove 1#1:f3c2593a:::obj26:head# 2024-09-18T15:59:20.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:767: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:59:20.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:59:20.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:59:20.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:59:20.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:59:20.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T15:59:20.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:59:20.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:59:20.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:59:20.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:59:20.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T15:59:20.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T15:59:20.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:59:20.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:59:20.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:59:20.560 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:59:20.560 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:59:20.560 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:59:20.560 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:59:20.560 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:59:20.560 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:59:20.561 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:59:20.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:59:20.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:59:20.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:59:20.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:59:20.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:59:20.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:59:20.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:59:20.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:59:20.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:59:20.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:59:20.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T15:59:20.563 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T15:59:20.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:59:20.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:59:20.565 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-18T15:59:20.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:59:20.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:59:20.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:59:20.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:59:20.597 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:59:20.588+0000 7f7dd9b49540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:59:20.600 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:59:20.592+0000 7f7dd9b49540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:59:20.603 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:59:20.592+0000 7f7dd9b49540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:59:20.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:59:20.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:59:20.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:59:20.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:59:20.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:59:20.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:59:20.876 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:59:20.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:59:20.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:59:20.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:59:21.137 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:59:21.128+0000 7f7dd9b49540 -1 Falling back to public interface 2024-09-18T15:59:21.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:59:22.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:59:22.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:59:22.227 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:59:22.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:59:22.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:59:22.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:59:22.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:59:23.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:59:23.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:59:23.538 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:59:23.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:59:23.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:59:23.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:59:23.758 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:59:23.748+0000 7f7dd9b49540 -1 osd.1 22 log_to_monitors true 2024-09-18T15:59:23.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:59:24.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:59:24.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:59:24.893 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:59:24.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:59:24.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:59:24.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:59:25.271 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [10,22) [v2:127.0.0.1:6802/460932,v1:127.0.0.1:6803/460932] [v2:127.0.0.1:6804/460932,v1:127.0.0.1:6805/460932] exists,up cbd0a5ba-dbbe-4416-8718-cef63e6222a8 2024-09-18T15:59:25.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:59:25.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:59:25.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:59:25.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:768: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:59:25.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T15:59:25.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:59:25.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:59:25.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:59:25.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:59:25.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:59:25.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:59:25.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:59:25.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:59:25.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:59:25.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:59:25.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:59:25.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:59:25.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:59:25.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:59:25.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:59:25.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:59:25.275 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:59:25.275 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:59:25.275 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:59:25.277 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:59:25.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:59:25.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:59:25.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:59:25.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:59:25.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:59:25.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:59:25.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:59:25.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:59:25.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:59:25.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-18T15:59:25.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:59:25.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:59:25.279 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T15:59:25.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-18T15:59:25.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T15:59:25.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:59:25.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:59:25.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:59:25.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:59:25.309 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:59:25.300+0000 7ff27e2a4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:59:25.334 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:59:25.324+0000 7ff27e2a4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:59:25.335 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:59:25.328+0000 7ff27e2a4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:59:25.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:59:25.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:59:25.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:59:25.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:59:25.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:59:25.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:59:25.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:59:25.623 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T15:59:25.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:59:25.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:59:25.877 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:59:25.868+0000 7ff27e2a4540 -1 Falling back to public interface 2024-09-18T15:59:25.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:59:26.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:59:26.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:59:26.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:59:26.962 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T15:59:26.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:59:26.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:59:27.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:59:28.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:59:28.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:59:28.279 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T15:59:28.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:59:28.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:59:28.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:59:28.539 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T15:59:28.532+0000 7ff27e2a4540 -1 osd.0 21 log_to_monitors true 2024-09-18T15:59:28.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:59:29.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:59:29.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:59:29.591 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T15:59:29.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:59:29.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:59:29.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:59:29.909 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6810/461510,v1:127.0.0.1:6811/461510] [v2:127.0.0.1:6812/461510,v1:127.0.0.1:6813/461510] exists,up afb8fc3c-6e91-47f8-a6ce-e33ad95d61fc 2024-09-18T15:59:29.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:59:29.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:59:29.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:59:29.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:769: TEST_repair_stats_ec: wait_for_clean 2024-09-18T15:59:29.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:59:29.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:59:29.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:59:29.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:59:29.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:59:29.911 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:59:29.911 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:59:29.912 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:59:29.912 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:59:29.912 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:59:30.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:59:30.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:59:30.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:59:30.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:59:30.147 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:59:30.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:59:30.488 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:59:30.488 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:59:30.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:59:30.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:59:30.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:59:30.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051586 2024-09-18T15:59:30.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051586 2024-09-18T15:59:30.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051586' 2024-09-18T15:59:30.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:59:30.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:59:30.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=111669149699 2024-09-18T15:59:30.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 111669149699 2024-09-18T15:59:30.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051586 1-111669149699' 2024-09-18T15:59:30.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:59:30.819 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:59:30.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509463 2024-09-18T15:59:30.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509463 2024-09-18T15:59:30.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051586 1-111669149699 2-64424509463' 2024-09-18T15:59:30.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:59:30.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-124554051586 2024-09-18T15:59:30.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:59:30.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:59:30.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-124554051586 2024-09-18T15:59:30.965 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:59:30.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051586 2024-09-18T15:59:30.967 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 124554051586 2024-09-18T15:59:30.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-09-18T15:59:30.968 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:59:31.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 124554051586 2024-09-18T15:59:31.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:59:32.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:59:32.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:59:32.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051586 -lt 124554051586 2024-09-18T15:59:32.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:59:32.592 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-111669149699 2024-09-18T15:59:32.593 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:59:32.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:59:32.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-111669149699 2024-09-18T15:59:32.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:59:32.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=111669149699 2024-09-18T15:59:32.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 111669149699' 2024-09-18T15:59:32.599 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 111669149699 2024-09-18T15:59:32.599 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:59:32.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 111669149699 -lt 111669149699 2024-09-18T15:59:32.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:59:32.942 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509463 2024-09-18T15:59:32.942 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:59:32.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:59:32.945 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509463 2024-09-18T15:59:32.946 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:59:32.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509463 2024-09-18T15:59:32.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-09-18T15:59:32.948 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509463 2024-09-18T15:59:32.948 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:59:33.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509463 -lt 64424509463 2024-09-18T15:59:33.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:59:33.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:59:33.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:59:33.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:59:33.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:59:33.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:59:33.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:59:33.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:59:33.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:59:33.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:59:33.659 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:59:33.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:59:33.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:59:33.986 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:59:33.986 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:59:34.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:59:34.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:59:34.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:59:34.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:771: TEST_repair_stats_ec: repair 1.0 2024-09-18T15:59:34.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=1.0 2024-09-18T15:59:34.369 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 1.0 2024-09-18T15:59:34.369 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:59:34.370 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:59:34.370 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:59:34.370 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:59:34.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T15:58:23.906080+0000 2024-09-18T15:59:34.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 1.0 2024-09-18T15:59:35.001 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-09-18T15:59:35.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 1.0 2024-09-18T15:58:23.906080+0000 2024-09-18T15:59:35.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-18T15:59:35.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T15:58:23.906080+0000 2024-09-18T15:59:35.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T15:59:35.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T15:59:35.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:59:35.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T15:59:35.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:59:35.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:59:35.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:59:35.015 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:59:35.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:58:23.906080+0000 '>' 2024-09-18T15:58:23.906080+0000 2024-09-18T15:59:35.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:59:36.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:59:36.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:59:36.311 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T15:59:36.311 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:59:36.311 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:59:36.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:59:36.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:59:36.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:58:23.906080+0000 '>' 2024-09-18T15:58:23.906080+0000 2024-09-18T15:59:36.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:59:37.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:59:37.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:59:37.625 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T15:59:37.625 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:59:37.625 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:59:37.626 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:59:37.626 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:59:37.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:58:23.906080+0000 '>' 2024-09-18T15:58:23.906080+0000 2024-09-18T15:59:37.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:59:38.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:59:38.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:59:38.940 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T15:59:38.940 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:59:38.940 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:59:38.941 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:59:38.941 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:59:39.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:58:23.906080+0000 '>' 2024-09-18T15:58:23.906080+0000 2024-09-18T15:59:39.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:59:40.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:59:40.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:59:40.259 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T15:59:40.259 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:59:40.259 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:59:40.260 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:59:40.260 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:59:40.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:58:23.906080+0000 '>' 2024-09-18T15:58:23.906080+0000 2024-09-18T15:59:40.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:59:41.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:59:41.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:59:41.571 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T15:59:41.571 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:59:41.571 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:59:41.572 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:59:41.572 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:59:41.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:58:23.906080+0000 '>' 2024-09-18T15:58:23.906080+0000 2024-09-18T15:59:41.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:59:42.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:59:42.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:59:42.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T15:59:42.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:59:42.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:59:42.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:59:42.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:59:43.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:58:23.906080+0000 '>' 2024-09-18T15:58:23.906080+0000 2024-09-18T15:59:43.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:59:44.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:59:44.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:59:44.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T15:59:44.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:59:44.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:59:44.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:59:44.209 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:59:44.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:58:23.906080+0000 '>' 2024-09-18T15:58:23.906080+0000 2024-09-18T15:59:44.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:59:45.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:59:45.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:59:45.521 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T15:59:45.521 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:59:45.521 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:59:45.522 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:59:45.522 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:59:45.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:58:23.906080+0000 '>' 2024-09-18T15:58:23.906080+0000 2024-09-18T15:59:45.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T15:59:46.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T15:59:46.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T15:59:46.837 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T15:59:46.837 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T15:59:46.838 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T15:59:46.838 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T15:59:46.838 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T15:59:47.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T15:59:39.070494+0000 '>' 2024-09-18T15:58:23.906080+0000 2024-09-18T15:59:47.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T15:59:47.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:772: TEST_repair_stats_ec: wait_for_clean 2024-09-18T15:59:47.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:59:47.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:59:47.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:59:47.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T15:59:47.149 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:59:47.150 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:59:47.150 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:59:47.151 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:59:47.151 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:59:47.151 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:59:47.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:59:47.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:59:47.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:59:47.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:59:47.397 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:59:47.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:59:47.712 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:59:47.712 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:59:47.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:59:47.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:59:47.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:59:47.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051590 2024-09-18T15:59:47.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051590 2024-09-18T15:59:47.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051590' 2024-09-18T15:59:47.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:59:47.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:59:48.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=111669149703 2024-09-18T15:59:48.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 111669149703 2024-09-18T15:59:48.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051590 1-111669149703' 2024-09-18T15:59:48.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:59:48.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:59:48.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509468 2024-09-18T15:59:48.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509468 2024-09-18T15:59:48.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051590 1-111669149703 2-64424509468' 2024-09-18T15:59:48.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:59:48.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-124554051590 2024-09-18T15:59:48.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:59:48.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:59:48.158 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-124554051590 2024-09-18T15:59:48.158 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:59:48.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051590 2024-09-18T15:59:48.161 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 124554051590 2024-09-18T15:59:48.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 124554051590' 2024-09-18T15:59:48.162 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:59:48.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051590 -lt 124554051590 2024-09-18T15:59:48.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:59:48.472 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-111669149703 2024-09-18T15:59:48.472 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:59:48.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:59:48.476 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-111669149703 2024-09-18T15:59:48.476 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:59:48.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=111669149703 2024-09-18T15:59:48.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 111669149703' 2024-09-18T15:59:48.478 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 111669149703 2024-09-18T15:59:48.479 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:59:48.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 111669149703 -lt 111669149703 2024-09-18T15:59:48.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:59:48.808 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509468 2024-09-18T15:59:48.808 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:59:48.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:59:48.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509468 2024-09-18T15:59:48.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:59:48.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509468 2024-09-18T15:59:48.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-09-18T15:59:48.816 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509468 2024-09-18T15:59:48.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:59:49.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509467 -lt 64424509468 2024-09-18T15:59:49.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:59:50.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:59:50.149 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:59:50.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509468 -lt 64424509468 2024-09-18T15:59:50.463 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:59:50.463 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:59:50.464 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:59:50.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:59:50.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:59:50.882 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:59:50.882 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:59:50.882 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:59:50.883 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:59:50.883 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:59:50.883 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:59:51.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:59:51.219 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:59:51.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:59:51.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:59:51.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:59:51.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:59:51.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:59:51.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:773: TEST_repair_stats_ec: ceph pg dump pgs 2024-09-18T15:59:51.912 INFO:tasks.workunit.client.0.smithi046.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T15:59:51.912 INFO:tasks.workunit.client.0.smithi046.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-09-18T15:59:40.416225+0000 21'30 30:133 [1,0,2] 1 [1,0,2] 1 21'30 2024-09-18T15:59:39.070494+0000 21'30 2024-09-18T15:59:39.070494+0000 0 1 periodic scrub scheduled @ 2024-09-19T15:59:39.070494+0000 17 0 2024-09-18T15:59:51.913 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-09-18T15:59:51.913 INFO:tasks.workunit.client.0.smithi046.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T15:59:51.915 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T15:59:51.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:774: TEST_repair_stats_ec: flush_pg_stats 2024-09-18T15:59:51.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:59:51.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:59:52.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:59:52.274 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T15:59:52.274 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T15:59:52.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:59:52.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:59:52.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:59:52.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051592 2024-09-18T15:59:52.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051592 2024-09-18T15:59:52.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051592' 2024-09-18T15:59:52.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:59:52.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:59:52.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=111669149705 2024-09-18T15:59:52.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 111669149705 2024-09-18T15:59:52.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051592 1-111669149705' 2024-09-18T15:59:52.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:59:52.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:59:52.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509470 2024-09-18T15:59:52.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509470 2024-09-18T15:59:52.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051592 1-111669149705 2-64424509470' 2024-09-18T15:59:52.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:59:52.739 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-124554051592 2024-09-18T15:59:52.740 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:59:52.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:59:52.744 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-124554051592 2024-09-18T15:59:52.744 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:59:52.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051592 2024-09-18T15:59:52.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 124554051592' 2024-09-18T15:59:52.747 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 124554051592 2024-09-18T15:59:52.748 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:59:53.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051591 -lt 124554051592 2024-09-18T15:59:53.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:59:54.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:59:54.048 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:59:54.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051593 -lt 124554051592 2024-09-18T15:59:54.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:59:54.364 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-111669149705 2024-09-18T15:59:54.365 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:59:54.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:59:54.369 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-111669149705 2024-09-18T15:59:54.369 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:59:54.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=111669149705 2024-09-18T15:59:54.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 111669149705' 2024-09-18T15:59:54.371 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 111669149705 2024-09-18T15:59:54.372 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:59:54.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 111669149705 -lt 111669149705 2024-09-18T15:59:54.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:59:54.702 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509470 2024-09-18T15:59:54.703 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:59:54.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:59:54.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509470 2024-09-18T15:59:54.707 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:59:54.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509470 2024-09-18T15:59:54.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509470' 2024-09-18T15:59:54.710 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509470 2024-09-18T15:59:54.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:59:55.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509470 -lt 64424509470 2024-09-18T15:59:55.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: ceph pg 1.0 query 2024-09-18T15:59:55.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: jq .info.stats.stat_sum 2024-09-18T15:59:55.219 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T15:59:55.219 INFO:tasks.workunit.client.0.smithi046.stdout: "num_bytes": 210, 2024-09-18T15:59:55.219 INFO:tasks.workunit.client.0.smithi046.stdout: "num_objects": 30, 2024-09-18T15:59:55.219 INFO:tasks.workunit.client.0.smithi046.stdout: "num_object_clones": 0, 2024-09-18T15:59:55.219 INFO:tasks.workunit.client.0.smithi046.stdout: "num_object_copies": 90, 2024-09-18T15:59:55.219 INFO:tasks.workunit.client.0.smithi046.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T15:59:55.220 INFO:tasks.workunit.client.0.smithi046.stdout: "num_objects_missing": 0, 2024-09-18T15:59:55.220 INFO:tasks.workunit.client.0.smithi046.stdout: "num_objects_degraded": 0, 2024-09-18T15:59:55.220 INFO:tasks.workunit.client.0.smithi046.stdout: "num_objects_misplaced": 0, 2024-09-18T15:59:55.220 INFO:tasks.workunit.client.0.smithi046.stdout: "num_objects_unfound": 0, 2024-09-18T15:59:55.220 INFO:tasks.workunit.client.0.smithi046.stdout: "num_objects_dirty": 30, 2024-09-18T15:59:55.220 INFO:tasks.workunit.client.0.smithi046.stdout: "num_whiteouts": 0, 2024-09-18T15:59:55.220 INFO:tasks.workunit.client.0.smithi046.stdout: "num_read": 0, 2024-09-18T15:59:55.220 INFO:tasks.workunit.client.0.smithi046.stdout: "num_read_kb": 0, 2024-09-18T15:59:55.220 INFO:tasks.workunit.client.0.smithi046.stdout: "num_write": 30, 2024-09-18T15:59:55.220 INFO:tasks.workunit.client.0.smithi046.stdout: "num_write_kb": 30, 2024-09-18T15:59:55.220 INFO:tasks.workunit.client.0.smithi046.stdout: "num_scrub_errors": 0, 2024-09-18T15:59:55.220 INFO:tasks.workunit.client.0.smithi046.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T15:59:55.220 INFO:tasks.workunit.client.0.smithi046.stdout: "num_deep_scrub_errors": 0, 2024-09-18T15:59:55.220 INFO:tasks.workunit.client.0.smithi046.stdout: "num_objects_recovered": 26, 2024-09-18T15:59:55.221 INFO:tasks.workunit.client.0.smithi046.stdout: "num_bytes_recovered": 182, 2024-09-18T15:59:55.221 INFO:tasks.workunit.client.0.smithi046.stdout: "num_keys_recovered": 0, 2024-09-18T15:59:55.221 INFO:tasks.workunit.client.0.smithi046.stdout: "num_objects_omap": 0, 2024-09-18T15:59:55.221 INFO:tasks.workunit.client.0.smithi046.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T15:59:55.221 INFO:tasks.workunit.client.0.smithi046.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T15:59:55.221 INFO:tasks.workunit.client.0.smithi046.stdout: "num_flush": 0, 2024-09-18T15:59:55.221 INFO:tasks.workunit.client.0.smithi046.stdout: "num_flush_kb": 0, 2024-09-18T15:59:55.221 INFO:tasks.workunit.client.0.smithi046.stdout: "num_evict": 0, 2024-09-18T15:59:55.221 INFO:tasks.workunit.client.0.smithi046.stdout: "num_evict_kb": 0, 2024-09-18T15:59:55.221 INFO:tasks.workunit.client.0.smithi046.stdout: "num_promote": 0, 2024-09-18T15:59:55.221 INFO:tasks.workunit.client.0.smithi046.stdout: "num_flush_mode_high": 0, 2024-09-18T15:59:55.221 INFO:tasks.workunit.client.0.smithi046.stdout: "num_flush_mode_low": 0, 2024-09-18T15:59:55.221 INFO:tasks.workunit.client.0.smithi046.stdout: "num_evict_mode_some": 0, 2024-09-18T15:59:55.222 INFO:tasks.workunit.client.0.smithi046.stdout: "num_evict_mode_full": 0, 2024-09-18T15:59:55.222 INFO:tasks.workunit.client.0.smithi046.stdout: "num_objects_pinned": 0, 2024-09-18T15:59:55.222 INFO:tasks.workunit.client.0.smithi046.stdout: "num_legacy_snapsets": 0, 2024-09-18T15:59:55.222 INFO:tasks.workunit.client.0.smithi046.stdout: "num_large_omap_objects": 0, 2024-09-18T15:59:55.222 INFO:tasks.workunit.client.0.smithi046.stdout: "num_objects_manifest": 0, 2024-09-18T15:59:55.222 INFO:tasks.workunit.client.0.smithi046.stdout: "num_omap_bytes": 0, 2024-09-18T15:59:55.222 INFO:tasks.workunit.client.0.smithi046.stdout: "num_omap_keys": 0, 2024-09-18T15:59:55.222 INFO:tasks.workunit.client.0.smithi046.stdout: "num_objects_repaired": 26 2024-09-18T15:59:55.222 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T15:59:55.223 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats_ec: ceph pg 1.0 query 2024-09-18T15:59:55.223 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats_ec: jq .info.stats.stat_sum.num_objects_repaired 2024-09-18T15:59:55.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats_ec: COUNT=26 2024-09-18T15:59:55.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:779: TEST_repair_stats_ec: test 26 = 26 2024-09-18T15:59:55.372 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: expr 3 - 1 2024-09-18T15:59:55.374 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: seq 0 2 2024-09-18T15:59:55.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T15:59:55.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: '[' 0 = 0 -o 0 = 1 ']' 2024-09-18T15:59:55.377 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:783: TEST_repair_stats_ec: expr 26 / 2 2024-09-18T15:59:55.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:783: TEST_repair_stats_ec: repair=13 2024-09-18T15:59:55.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-18T15:59:55.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-09-18T15:59:55.675 INFO:tasks.workunit.client.0.smithi046.stderr:dumped all 2024-09-18T15:59:55.683 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T15:59:55.683 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T15:59:55.683 INFO:tasks.workunit.client.0.smithi046.stdout: "up_from": 29, 2024-09-18T15:59:55.684 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 124554051593, 2024-09-18T15:59:55.684 INFO:tasks.workunit.client.0.smithi046.stdout: "num_pgs": 1, 2024-09-18T15:59:55.684 INFO:tasks.workunit.client.0.smithi046.stdout: "num_osds": 1, 2024-09-18T15:59:55.684 INFO:tasks.workunit.client.0.smithi046.stdout: "num_per_pool_osds": 1, 2024-09-18T15:59:55.684 INFO:tasks.workunit.client.0.smithi046.stdout: "num_per_pool_omap_osds": 1, 2024-09-18T15:59:55.684 INFO:tasks.workunit.client.0.smithi046.stdout: "kb": 104857600, 2024-09-18T15:59:55.684 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used": 298168, 2024-09-18T15:59:55.684 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used_data": 368, 2024-09-18T15:59:55.684 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used_omap": 4, 2024-09-18T15:59:55.684 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used_meta": 297787, 2024-09-18T15:59:55.684 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_avail": 104559432, 2024-09-18T15:59:55.684 INFO:tasks.workunit.client.0.smithi046.stdout: "statfs": { 2024-09-18T15:59:55.684 INFO:tasks.workunit.client.0.smithi046.stdout: "total": 107374182400, 2024-09-18T15:59:55.684 INFO:tasks.workunit.client.0.smithi046.stdout: "available": 107068858368, 2024-09-18T15:59:55.685 INFO:tasks.workunit.client.0.smithi046.stdout: "internally_reserved": 0, 2024-09-18T15:59:55.685 INFO:tasks.workunit.client.0.smithi046.stdout: "allocated": 376832, 2024-09-18T15:59:55.685 INFO:tasks.workunit.client.0.smithi046.stdout: "data_stored": 201435, 2024-09-18T15:59:55.685 INFO:tasks.workunit.client.0.smithi046.stdout: "data_compressed": 0, 2024-09-18T15:59:55.685 INFO:tasks.workunit.client.0.smithi046.stdout: "data_compressed_allocated": 0, 2024-09-18T15:59:55.685 INFO:tasks.workunit.client.0.smithi046.stdout: "data_compressed_original": 0, 2024-09-18T15:59:55.685 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_allocated": 4543, 2024-09-18T15:59:55.685 INFO:tasks.workunit.client.0.smithi046.stdout: "internal_metadata": 304934465 2024-09-18T15:59:55.685 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:55.685 INFO:tasks.workunit.client.0.smithi046.stdout: "hb_peers": [ 2024-09-18T15:59:55.685 INFO:tasks.workunit.client.0.smithi046.stdout: 1, 2024-09-18T15:59:55.685 INFO:tasks.workunit.client.0.smithi046.stdout: 2 2024-09-18T15:59:55.685 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T15:59:55.686 INFO:tasks.workunit.client.0.smithi046.stdout: "snap_trim_queue_len": 0, 2024-09-18T15:59:55.686 INFO:tasks.workunit.client.0.smithi046.stdout: "num_snap_trimming": 0, 2024-09-18T15:59:55.686 INFO:tasks.workunit.client.0.smithi046.stdout: "num_shards_repaired": 13, 2024-09-18T15:59:55.686 INFO:tasks.workunit.client.0.smithi046.stdout: "op_queue_age_hist": { 2024-09-18T15:59:55.686 INFO:tasks.workunit.client.0.smithi046.stdout: "histogram": [], 2024-09-18T15:59:55.686 INFO:tasks.workunit.client.0.smithi046.stdout: "upper_bound": 1 2024-09-18T15:59:55.686 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:55.686 INFO:tasks.workunit.client.0.smithi046.stdout: "perf_stat": { 2024-09-18T15:59:55.686 INFO:tasks.workunit.client.0.smithi046.stdout: "commit_latency_ms": 0, 2024-09-18T15:59:55.686 INFO:tasks.workunit.client.0.smithi046.stdout: "apply_latency_ms": 0, 2024-09-18T15:59:55.686 INFO:tasks.workunit.client.0.smithi046.stdout: "commit_latency_ns": 0, 2024-09-18T15:59:55.686 INFO:tasks.workunit.client.0.smithi046.stdout: "apply_latency_ns": 0 2024-09-18T15:59:55.686 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:55.686 INFO:tasks.workunit.client.0.smithi046.stdout: "alerts": [], 2024-09-18T15:59:55.686 INFO:tasks.workunit.client.0.smithi046.stdout: "network_ping_times": [ 2024-09-18T15:59:55.687 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T15:59:55.687 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T15:59:55.687 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2024-09-18T15:59:55.687 INFO:tasks.workunit.client.0.smithi046.stdout: "interfaces": [ 2024-09-18T15:59:55.687 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T15:59:55.687 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-09-18T15:59:55.687 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-09-18T15:59:55.687 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0, 2024-09-18T15:59:55.687 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0, 2024-09-18T15:59:55.687 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0 2024-09-18T15:59:55.687 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:55.687 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-09-18T15:59:55.687 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0, 2024-09-18T15:59:55.687 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0, 2024-09-18T15:59:55.687 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0 2024-09-18T15:59:55.688 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:55.688 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-09-18T15:59:55.688 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0, 2024-09-18T15:59:55.688 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0, 2024-09-18T15:59:55.688 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0 2024-09-18T15:59:55.688 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:55.688 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.512 2024-09-18T15:59:55.688 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T15:59:55.688 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T15:59:55.688 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:55.688 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T15:59:55.688 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 2, 2024-09-18T15:59:55.688 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2024-09-18T15:59:55.688 INFO:tasks.workunit.client.0.smithi046.stdout: "interfaces": [ 2024-09-18T15:59:55.689 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T15:59:55.689 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-09-18T15:59:55.689 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-09-18T15:59:55.689 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0, 2024-09-18T15:59:55.689 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0, 2024-09-18T15:59:55.689 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0 2024-09-18T15:59:55.689 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:55.689 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-09-18T15:59:55.689 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0, 2024-09-18T15:59:55.689 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0, 2024-09-18T15:59:55.689 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0 2024-09-18T15:59:55.689 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:55.689 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-09-18T15:59:55.689 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0, 2024-09-18T15:59:55.689 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0, 2024-09-18T15:59:55.690 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0 2024-09-18T15:59:55.690 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:55.690 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.605 2024-09-18T15:59:55.690 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T15:59:55.690 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T15:59:55.690 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T15:59:55.690 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T15:59:55.690 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T15:59:55.690 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-18T15:59:55.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-09-18T15:59:55.973 INFO:tasks.workunit.client.0.smithi046.stderr:dumped all 2024-09-18T15:59:55.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: COUNT=13 2024-09-18T15:59:55.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:790: TEST_repair_stats_ec: test 13 = 13 2024-09-18T15:59:55.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T15:59:55.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: '[' 1 = 0 -o 1 = 1 ']' 2024-09-18T15:59:55.984 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:783: TEST_repair_stats_ec: expr 26 / 2 2024-09-18T15:59:55.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:783: TEST_repair_stats_ec: repair=13 2024-09-18T15:59:55.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-18T15:59:55.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2024-09-18T15:59:56.293 INFO:tasks.workunit.client.0.smithi046.stderr:dumped all 2024-09-18T15:59:56.301 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T15:59:56.301 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T15:59:56.301 INFO:tasks.workunit.client.0.smithi046.stdout: "up_from": 26, 2024-09-18T15:59:56.301 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 111669149706, 2024-09-18T15:59:56.301 INFO:tasks.workunit.client.0.smithi046.stdout: "num_pgs": 1, 2024-09-18T15:59:56.301 INFO:tasks.workunit.client.0.smithi046.stdout: "num_osds": 1, 2024-09-18T15:59:56.301 INFO:tasks.workunit.client.0.smithi046.stdout: "num_per_pool_osds": 1, 2024-09-18T15:59:56.302 INFO:tasks.workunit.client.0.smithi046.stdout: "num_per_pool_omap_osds": 1, 2024-09-18T15:59:56.302 INFO:tasks.workunit.client.0.smithi046.stdout: "kb": 104857600, 2024-09-18T15:59:56.302 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used": 298232, 2024-09-18T15:59:56.302 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used_data": 368, 2024-09-18T15:59:56.302 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used_omap": 4, 2024-09-18T15:59:56.302 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used_meta": 297851, 2024-09-18T15:59:56.302 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_avail": 104559368, 2024-09-18T15:59:56.302 INFO:tasks.workunit.client.0.smithi046.stdout: "statfs": { 2024-09-18T15:59:56.302 INFO:tasks.workunit.client.0.smithi046.stdout: "total": 107374182400, 2024-09-18T15:59:56.302 INFO:tasks.workunit.client.0.smithi046.stdout: "available": 107068792832, 2024-09-18T15:59:56.302 INFO:tasks.workunit.client.0.smithi046.stdout: "internally_reserved": 0, 2024-09-18T15:59:56.302 INFO:tasks.workunit.client.0.smithi046.stdout: "allocated": 376832, 2024-09-18T15:59:56.302 INFO:tasks.workunit.client.0.smithi046.stdout: "data_stored": 201435, 2024-09-18T15:59:56.302 INFO:tasks.workunit.client.0.smithi046.stdout: "data_compressed": 0, 2024-09-18T15:59:56.303 INFO:tasks.workunit.client.0.smithi046.stdout: "data_compressed_allocated": 0, 2024-09-18T15:59:56.303 INFO:tasks.workunit.client.0.smithi046.stdout: "data_compressed_original": 0, 2024-09-18T15:59:56.303 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_allocated": 4543, 2024-09-18T15:59:56.303 INFO:tasks.workunit.client.0.smithi046.stdout: "internal_metadata": 305000001 2024-09-18T15:59:56.303 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:56.303 INFO:tasks.workunit.client.0.smithi046.stdout: "hb_peers": [ 2024-09-18T15:59:56.303 INFO:tasks.workunit.client.0.smithi046.stdout: 0, 2024-09-18T15:59:56.303 INFO:tasks.workunit.client.0.smithi046.stdout: 2 2024-09-18T15:59:56.303 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T15:59:56.303 INFO:tasks.workunit.client.0.smithi046.stdout: "snap_trim_queue_len": 0, 2024-09-18T15:59:56.303 INFO:tasks.workunit.client.0.smithi046.stdout: "num_snap_trimming": 0, 2024-09-18T15:59:56.303 INFO:tasks.workunit.client.0.smithi046.stdout: "num_shards_repaired": 13, 2024-09-18T15:59:56.304 INFO:tasks.workunit.client.0.smithi046.stdout: "op_queue_age_hist": { 2024-09-18T15:59:56.304 INFO:tasks.workunit.client.0.smithi046.stdout: "histogram": [], 2024-09-18T15:59:56.304 INFO:tasks.workunit.client.0.smithi046.stdout: "upper_bound": 1 2024-09-18T15:59:56.304 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:56.304 INFO:tasks.workunit.client.0.smithi046.stdout: "perf_stat": { 2024-09-18T15:59:56.304 INFO:tasks.workunit.client.0.smithi046.stdout: "commit_latency_ms": 0, 2024-09-18T15:59:56.304 INFO:tasks.workunit.client.0.smithi046.stdout: "apply_latency_ms": 0, 2024-09-18T15:59:56.304 INFO:tasks.workunit.client.0.smithi046.stdout: "commit_latency_ns": 0, 2024-09-18T15:59:56.304 INFO:tasks.workunit.client.0.smithi046.stdout: "apply_latency_ns": 0 2024-09-18T15:59:56.304 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:56.304 INFO:tasks.workunit.client.0.smithi046.stdout: "alerts": [], 2024-09-18T15:59:56.304 INFO:tasks.workunit.client.0.smithi046.stdout: "network_ping_times": [ 2024-09-18T15:59:56.304 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T15:59:56.304 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T15:59:56.305 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2024-09-18T15:59:56.305 INFO:tasks.workunit.client.0.smithi046.stdout: "interfaces": [ 2024-09-18T15:59:56.305 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T15:59:56.305 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-09-18T15:59:56.305 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-09-18T15:59:56.305 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0, 2024-09-18T15:59:56.305 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0, 2024-09-18T15:59:56.305 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0 2024-09-18T15:59:56.305 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:56.305 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-09-18T15:59:56.305 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0, 2024-09-18T15:59:56.305 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0, 2024-09-18T15:59:56.305 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0 2024-09-18T15:59:56.306 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:56.306 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-09-18T15:59:56.306 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0, 2024-09-18T15:59:56.306 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0, 2024-09-18T15:59:56.306 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0 2024-09-18T15:59:56.306 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:56.306 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.295 2024-09-18T15:59:56.306 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T15:59:56.306 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T15:59:56.307 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:56.307 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T15:59:56.307 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 2, 2024-09-18T15:59:56.307 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2024-09-18T15:59:56.307 INFO:tasks.workunit.client.0.smithi046.stdout: "interfaces": [ 2024-09-18T15:59:56.307 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T15:59:56.307 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-09-18T15:59:56.307 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-09-18T15:59:56.307 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0, 2024-09-18T15:59:56.307 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0, 2024-09-18T15:59:56.307 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0 2024-09-18T15:59:56.307 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:56.307 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-09-18T15:59:56.308 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0, 2024-09-18T15:59:56.308 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0, 2024-09-18T15:59:56.308 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0 2024-09-18T15:59:56.308 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:56.308 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-09-18T15:59:56.308 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0, 2024-09-18T15:59:56.308 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0, 2024-09-18T15:59:56.308 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0 2024-09-18T15:59:56.308 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:56.308 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.399 2024-09-18T15:59:56.308 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T15:59:56.308 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T15:59:56.308 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T15:59:56.308 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T15:59:56.309 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T15:59:56.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-18T15:59:56.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2024-09-18T15:59:56.596 INFO:tasks.workunit.client.0.smithi046.stderr:dumped all 2024-09-18T15:59:56.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: COUNT=13 2024-09-18T15:59:56.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:790: TEST_repair_stats_ec: test 13 = 13 2024-09-18T15:59:56.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T15:59:56.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: '[' 2 = 0 -o 2 = 1 ']' 2024-09-18T15:59:56.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:785: TEST_repair_stats_ec: repair=0 2024-09-18T15:59:56.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-18T15:59:56.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 2 )' 2024-09-18T15:59:56.893 INFO:tasks.workunit.client.0.smithi046.stderr:dumped all 2024-09-18T15:59:56.901 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T15:59:56.901 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 2, 2024-09-18T15:59:56.901 INFO:tasks.workunit.client.0.smithi046.stdout: "up_from": 15, 2024-09-18T15:59:56.902 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 64424509470, 2024-09-18T15:59:56.902 INFO:tasks.workunit.client.0.smithi046.stdout: "num_pgs": 1, 2024-09-18T15:59:56.902 INFO:tasks.workunit.client.0.smithi046.stdout: "num_osds": 1, 2024-09-18T15:59:56.902 INFO:tasks.workunit.client.0.smithi046.stdout: "num_per_pool_osds": 1, 2024-09-18T15:59:56.902 INFO:tasks.workunit.client.0.smithi046.stdout: "num_per_pool_omap_osds": 1, 2024-09-18T15:59:56.902 INFO:tasks.workunit.client.0.smithi046.stdout: "kb": 104857600, 2024-09-18T15:59:56.902 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used": 297532, 2024-09-18T15:59:56.902 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used_data": 364, 2024-09-18T15:59:56.902 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used_omap": 1, 2024-09-18T15:59:56.902 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used_meta": 297150, 2024-09-18T15:59:56.902 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_avail": 104560068, 2024-09-18T15:59:56.902 INFO:tasks.workunit.client.0.smithi046.stdout: "statfs": { 2024-09-18T15:59:56.902 INFO:tasks.workunit.client.0.smithi046.stdout: "total": 107374182400, 2024-09-18T15:59:56.903 INFO:tasks.workunit.client.0.smithi046.stdout: "available": 107069509632, 2024-09-18T15:59:56.903 INFO:tasks.workunit.client.0.smithi046.stdout: "internally_reserved": 0, 2024-09-18T15:59:56.903 INFO:tasks.workunit.client.0.smithi046.stdout: "allocated": 372736, 2024-09-18T15:59:56.903 INFO:tasks.workunit.client.0.smithi046.stdout: "data_stored": 201191, 2024-09-18T15:59:56.903 INFO:tasks.workunit.client.0.smithi046.stdout: "data_compressed": 0, 2024-09-18T15:59:56.903 INFO:tasks.workunit.client.0.smithi046.stdout: "data_compressed_allocated": 0, 2024-09-18T15:59:56.903 INFO:tasks.workunit.client.0.smithi046.stdout: "data_compressed_original": 0, 2024-09-18T15:59:56.903 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_allocated": 1704, 2024-09-18T15:59:56.903 INFO:tasks.workunit.client.0.smithi046.stdout: "internal_metadata": 304281944 2024-09-18T15:59:56.903 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:56.903 INFO:tasks.workunit.client.0.smithi046.stdout: "hb_peers": [ 2024-09-18T15:59:56.903 INFO:tasks.workunit.client.0.smithi046.stdout: 0, 2024-09-18T15:59:56.903 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T15:59:56.904 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T15:59:56.904 INFO:tasks.workunit.client.0.smithi046.stdout: "snap_trim_queue_len": 0, 2024-09-18T15:59:56.904 INFO:tasks.workunit.client.0.smithi046.stdout: "num_snap_trimming": 0, 2024-09-18T15:59:56.904 INFO:tasks.workunit.client.0.smithi046.stdout: "num_shards_repaired": 0, 2024-09-18T15:59:56.904 INFO:tasks.workunit.client.0.smithi046.stdout: "op_queue_age_hist": { 2024-09-18T15:59:56.904 INFO:tasks.workunit.client.0.smithi046.stdout: "histogram": [], 2024-09-18T15:59:56.904 INFO:tasks.workunit.client.0.smithi046.stdout: "upper_bound": 1 2024-09-18T15:59:56.904 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:56.904 INFO:tasks.workunit.client.0.smithi046.stdout: "perf_stat": { 2024-09-18T15:59:56.904 INFO:tasks.workunit.client.0.smithi046.stdout: "commit_latency_ms": 0, 2024-09-18T15:59:56.904 INFO:tasks.workunit.client.0.smithi046.stdout: "apply_latency_ms": 0, 2024-09-18T15:59:56.904 INFO:tasks.workunit.client.0.smithi046.stdout: "commit_latency_ns": 0, 2024-09-18T15:59:56.904 INFO:tasks.workunit.client.0.smithi046.stdout: "apply_latency_ns": 0 2024-09-18T15:59:56.905 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:56.905 INFO:tasks.workunit.client.0.smithi046.stdout: "alerts": [], 2024-09-18T15:59:56.905 INFO:tasks.workunit.client.0.smithi046.stdout: "network_ping_times": [ 2024-09-18T15:59:56.905 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T15:59:56.905 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T15:59:56.905 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2024-09-18T15:59:56.905 INFO:tasks.workunit.client.0.smithi046.stdout: "interfaces": [ 2024-09-18T15:59:56.905 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T15:59:56.905 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-09-18T15:59:56.905 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-09-18T15:59:56.905 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0, 2024-09-18T15:59:56.905 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0, 2024-09-18T15:59:56.905 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0 2024-09-18T15:59:56.906 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:56.906 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-09-18T15:59:56.906 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0, 2024-09-18T15:59:56.906 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0, 2024-09-18T15:59:56.906 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0 2024-09-18T15:59:56.906 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:56.906 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-09-18T15:59:56.906 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0, 2024-09-18T15:59:56.906 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0, 2024-09-18T15:59:56.906 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0 2024-09-18T15:59:56.906 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:56.906 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.447 2024-09-18T15:59:56.906 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T15:59:56.906 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T15:59:56.907 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:56.907 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T15:59:56.907 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T15:59:56.907 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2024-09-18T15:59:56.907 INFO:tasks.workunit.client.0.smithi046.stdout: "interfaces": [ 2024-09-18T15:59:56.907 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T15:59:56.907 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-09-18T15:59:56.907 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-09-18T15:59:56.907 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0, 2024-09-18T15:59:56.907 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0, 2024-09-18T15:59:56.907 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0 2024-09-18T15:59:56.907 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:56.907 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-09-18T15:59:56.907 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0, 2024-09-18T15:59:56.907 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0, 2024-09-18T15:59:56.908 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0 2024-09-18T15:59:56.908 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:56.908 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-09-18T15:59:56.908 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0, 2024-09-18T15:59:56.908 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0, 2024-09-18T15:59:56.908 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0 2024-09-18T15:59:56.908 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:56.908 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.538 2024-09-18T15:59:56.908 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T15:59:56.908 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T15:59:56.908 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T15:59:56.908 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T15:59:56.908 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T15:59:56.909 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-18T15:59:56.909 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 2 ).num_shards_repaired' 2024-09-18T15:59:57.188 INFO:tasks.workunit.client.0.smithi046.stderr:dumped all 2024-09-18T15:59:57.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: COUNT=0 2024-09-18T15:59:57.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:790: TEST_repair_stats_ec: test 0 = 0 2024-09-18T15:59:57.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:793: TEST_repair_stats_ec: jq .pg_map.osd_stats_sum 2024-09-18T15:59:57.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:793: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-18T15:59:57.486 INFO:tasks.workunit.client.0.smithi046.stderr:dumped all 2024-09-18T15:59:57.495 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T15:59:57.495 INFO:tasks.workunit.client.0.smithi046.stdout: "up_from": 0, 2024-09-18T15:59:57.495 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 0, 2024-09-18T15:59:57.495 INFO:tasks.workunit.client.0.smithi046.stdout: "num_pgs": 3, 2024-09-18T15:59:57.495 INFO:tasks.workunit.client.0.smithi046.stdout: "num_osds": 3, 2024-09-18T15:59:57.495 INFO:tasks.workunit.client.0.smithi046.stdout: "num_per_pool_osds": 3, 2024-09-18T15:59:57.495 INFO:tasks.workunit.client.0.smithi046.stdout: "num_per_pool_omap_osds": 3, 2024-09-18T15:59:57.495 INFO:tasks.workunit.client.0.smithi046.stdout: "kb": 314572800, 2024-09-18T15:59:57.495 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used": 893932, 2024-09-18T15:59:57.495 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used_data": 1100, 2024-09-18T15:59:57.495 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used_omap": 10, 2024-09-18T15:59:57.495 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_used_meta": 892789, 2024-09-18T15:59:57.495 INFO:tasks.workunit.client.0.smithi046.stdout: "kb_avail": 313678868, 2024-09-18T15:59:57.496 INFO:tasks.workunit.client.0.smithi046.stdout: "statfs": { 2024-09-18T15:59:57.496 INFO:tasks.workunit.client.0.smithi046.stdout: "total": 322122547200, 2024-09-18T15:59:57.496 INFO:tasks.workunit.client.0.smithi046.stdout: "available": 321207160832, 2024-09-18T15:59:57.496 INFO:tasks.workunit.client.0.smithi046.stdout: "internally_reserved": 0, 2024-09-18T15:59:57.496 INFO:tasks.workunit.client.0.smithi046.stdout: "allocated": 1126400, 2024-09-18T15:59:57.496 INFO:tasks.workunit.client.0.smithi046.stdout: "data_stored": 604061, 2024-09-18T15:59:57.496 INFO:tasks.workunit.client.0.smithi046.stdout: "data_compressed": 0, 2024-09-18T15:59:57.496 INFO:tasks.workunit.client.0.smithi046.stdout: "data_compressed_allocated": 0, 2024-09-18T15:59:57.496 INFO:tasks.workunit.client.0.smithi046.stdout: "data_compressed_original": 0, 2024-09-18T15:59:57.496 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_allocated": 10790, 2024-09-18T15:59:57.496 INFO:tasks.workunit.client.0.smithi046.stdout: "internal_metadata": 914216410 2024-09-18T15:59:57.496 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:57.496 INFO:tasks.workunit.client.0.smithi046.stdout: "hb_peers": [], 2024-09-18T15:59:57.496 INFO:tasks.workunit.client.0.smithi046.stdout: "snap_trim_queue_len": 0, 2024-09-18T15:59:57.497 INFO:tasks.workunit.client.0.smithi046.stdout: "num_snap_trimming": 0, 2024-09-18T15:59:57.497 INFO:tasks.workunit.client.0.smithi046.stdout: "num_shards_repaired": 26, 2024-09-18T15:59:57.497 INFO:tasks.workunit.client.0.smithi046.stdout: "op_queue_age_hist": { 2024-09-18T15:59:57.497 INFO:tasks.workunit.client.0.smithi046.stdout: "histogram": [], 2024-09-18T15:59:57.497 INFO:tasks.workunit.client.0.smithi046.stdout: "upper_bound": 1 2024-09-18T15:59:57.497 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:57.497 INFO:tasks.workunit.client.0.smithi046.stdout: "perf_stat": { 2024-09-18T15:59:57.497 INFO:tasks.workunit.client.0.smithi046.stdout: "commit_latency_ms": 0, 2024-09-18T15:59:57.497 INFO:tasks.workunit.client.0.smithi046.stdout: "apply_latency_ms": 0, 2024-09-18T15:59:57.497 INFO:tasks.workunit.client.0.smithi046.stdout: "commit_latency_ns": 0, 2024-09-18T15:59:57.497 INFO:tasks.workunit.client.0.smithi046.stdout: "apply_latency_ns": 0 2024-09-18T15:59:57.497 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T15:59:57.497 INFO:tasks.workunit.client.0.smithi046.stdout: "alerts": [], 2024-09-18T15:59:57.497 INFO:tasks.workunit.client.0.smithi046.stdout: "network_ping_times": [] 2024-09-18T15:59:57.497 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T15:59:57.498 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:794: TEST_repair_stats_ec: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-09-18T15:59:57.498 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:794: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-18T15:59:57.789 INFO:tasks.workunit.client.0.smithi046.stderr:dumped all 2024-09-18T15:59:57.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:794: TEST_repair_stats_ec: COUNT=26 2024-09-18T15:59:57.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:795: TEST_repair_stats_ec: test 26 = 26 2024-09-18T15:59:57.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T15:59:57.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:59:57.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:59:57.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:59:57.800 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:59:57.800 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:59:57.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:59:57.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:59:57.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:59:57.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:59:57.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:59:57.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:59:57.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:59:57.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:59:57.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:59:57.926 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:59:57.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:59:57.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:59:57.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:59:57.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:59:57.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:59:57.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:59:57.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:59:57.954 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:59:57.954 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:59:57.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:59:57.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:59:57.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:59:57.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:59:57.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T15:59:57.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T15:59:57.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T15:59:57.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T15:59:57.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T15:59:57.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:59:57.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T15:59:57.958 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:59:57.958 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:59:57.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:59:57.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:59:57.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:59:57.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:59:57.965 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:59:57.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:59:57.967 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:59:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:59:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:59:57.970 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:59:57.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:59:57.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:59:57.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:59:57.974 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:59:57.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:59:57.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:59:57.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T15:59:57.980 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:59:57.980 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:59:57.980 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:59:57.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T15:59:57.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:59:57.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:59:57.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T15:59:57.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:59:57.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:59:57.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:59:57.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T15:59:57.989 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:59:57.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:59:57.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:59:57.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T15:59:57.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_request_scrub_priority td/osd-scrub-repair 2024-09-18T15:59:57.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6218: TEST_request_scrub_priority: local dir=td/osd-scrub-repair 2024-09-18T15:59:57.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: local poolname=psr_pool 2024-09-18T15:59:57.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6220: TEST_request_scrub_priority: local objname=POBJ 2024-09-18T15:59:57.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: local OBJECTS=64 2024-09-18T15:59:57.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: local PGS=8 2024-09-18T15:59:57.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6224: TEST_request_scrub_priority: run_mon td/osd-scrub-repair a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T15:59:57.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T15:59:57.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:59:57.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:59:57.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:59:57.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T15:59:57.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T15:59:58.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:59:58.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:59:58.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:59:58.362 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:59:58.362 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:59:58.362 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:59:58.362 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:59:58.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T15:59:58.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:59:58.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:59:58.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:59:58.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:59:58.644 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:59:58.644 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:59:58.645 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:59:58.645 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:59:58.645 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:59:58.646 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:59:58.646 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:59:58.646 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:59:58.647 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:59:58.648 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:59:58.648 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T15:59:58.744 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:59:58.744 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:59:58.745 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:59:58.745 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:59:58.745 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:59:58.745 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:59:58.745 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:59:58.746 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:59:58.746 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:59:58.746 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:59:58.746 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:59:58.746 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T15:59:58.747 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:59:58.747 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T15:59:58.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: run_mgr td/osd-scrub-repair x 2024-09-18T15:59:58.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T15:59:58.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:59:58.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:59:58.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:59:58.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T15:59:58.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:59:59.687 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:59:59.687 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:59:59.688 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:59:59.688 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:59:59.688 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:59:59.688 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:59:59.689 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:59:59.690 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:59:59.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:59:59.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2024-09-18T15:59:59.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6227: TEST_request_scrub_priority: ceph_osd_args+=--osd_scrub_backoff_ratio=0 2024-09-18T15:59:59.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-09-18T15:59:59.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T15:59:59.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:59:59.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:59:59.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:59:59.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T15:59:59.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T15:59:59.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:59:59.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:59:59.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:59:59.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T15:59:59.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T15:59:59.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:59:59.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:59:59.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T15:59:59.727 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:59:59.727 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:59:59.727 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:59:59.728 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:59:59.728 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:59:59.728 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T15:59:59.728 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:59:59.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T15:59:59.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:59:59.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:59:59.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:59:59.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T15:59:59.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T15:59:59.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:59:59.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:59:59.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:59:59.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:59:59.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:59:59.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-09-18T15:59:59.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T15:59:59.729 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:59:59.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=558f3228-0e38-4f81-9005-c1d52476dda0 2024-09-18T15:59:59.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 558f3228-0e38-4f81-9005-c1d52476dda0' 2024-09-18T15:59:59.731 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 558f3228-0e38-4f81-9005-c1d52476dda0 2024-09-18T15:59:59.731 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:59:59.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD/+OpmACxuLBAA1AXK1/gFx3LfvOiKEE6RyA== 2024-09-18T15:59:59.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD/+OpmACxuLBAA1AXK1/gFx3LfvOiKEE6RyA=="}' 2024-09-18T15:59:59.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 558f3228-0e38-4f81-9005-c1d52476dda0 -i td/osd-scrub-repair/0/new.json 2024-09-18T16:00:00.035 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:00:00.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-18T16:00:00.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQD/+OpmACxuLBAA1AXK1/gFx3LfvOiKEE6RyA== --osd-uuid 558f3228-0e38-4f81-9005-c1d52476dda0 2024-09-18T16:00:00.087 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:00:00.083+0000 7fb21c880540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:00:00.090 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:00:00.083+0000 7fb21c880540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:00:00.092 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:00:00.087+0000 7fb21c880540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:00:00.094 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:00:00.087+0000 7fb21c880540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T16:00:00.094 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:00:00.087+0000 7fb21c880540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T16:00:00.094 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:00:00.087+0000 7fb21c880540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T16:00:00.167 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:00:00.159+0000 7fb21c880540 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-18T16:00:06.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T16:00:06.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:00:06.362 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T16:00:06.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T16:00:06.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:00:06.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T16:00:06.750 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T16:00:06.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-09-18T16:00:06.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:00:06.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:00:06.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:00:06.783 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:00:06.779+0000 7fe7b2f97540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:00:06.786 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:00:06.779+0000 7fe7b2f97540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:00:06.787 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:00:06.783+0000 7fe7b2f97540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:00:07.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T16:00:07.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:00:07.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:00:07.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:00:07.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:00:07.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:07.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:00:07.094 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:00:07.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:07.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:07.321 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:00:07.315+0000 7fe7b2f97540 -1 Falling back to public interface 2024-09-18T16:00:07.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:08.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:08.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:08.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:00:08.389 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:00:08.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:08.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:08.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:09.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:09.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:09.700 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:00:09.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:00:09.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:09.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:10.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:10.019 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:00:10.015+0000 7fe7b2f97540 -1 osd.0 0 log_to_monitors true 2024-09-18T16:00:11.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:11.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:11.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:00:11.011 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:00:11.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:11.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:11.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:12.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:12.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:12.332 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:00:12.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:00:12.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:12.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:12.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:13.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:13.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:13.652 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:00:13.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:00:13.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:13.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:13.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:14.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:14.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:14.989 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:00:14.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:00:14.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:14.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:15.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:16.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:16.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:16.314 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:00:16.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:00:16.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:16.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:16.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:17.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:17.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:17.635 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:00:17.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:00:17.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:17.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:17.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:18.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:18.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:18.999 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:00:18.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:00:18.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:18.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:19.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:20.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:20.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:20.317 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:00:20.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:00:20.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:20.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:20.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:21.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:21.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:21.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:00:21.624 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:00:21.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:21.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:22.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:23.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:23.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:23.007 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:00:23.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:00:23.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:23.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:23.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:24.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:24.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:24.485 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:00:24.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:00:24.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:24.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:25.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:26.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:26.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:26.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:00:26.021 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:00:26.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:26.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:26.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:27.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:27.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:27.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:00:27.591 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:00:27.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:27.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:28.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:29.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:29.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:29.270 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:00:29.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:00:29.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:29.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:29.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:30.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:30.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:30.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:00:30.833 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:00:30.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:30.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:31.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:32.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:32.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:32.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:00:32.498 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:00:32.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:32.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:32.812 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/464765,v1:127.0.0.1:6803/464765] [v2:127.0.0.1:6804/464765,v1:127.0.0.1:6805/464765] exists,up 558f3228-0e38-4f81-9005-c1d52476dda0 2024-09-18T16:00:32.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:00:32.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:00:32.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:00:32.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6230: TEST_request_scrub_priority: create_pool psr_pool 8 8 2024-09-18T16:00:32.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool 8 8 2024-09-18T16:00:33.413 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'psr_pool' created 2024-09-18T16:00:33.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T16:00:34.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: wait_for_clean 2024-09-18T16:00:34.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:00:34.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:00:34.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:00:34.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:00:34.429 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:00:34.430 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:00:34.430 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:00:34.431 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:00:34.431 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:00:34.431 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:00:34.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:00:34.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:00:34.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:00:34.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:00:34.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:00:34.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T16:00:34.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:00:34.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:00:34.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:00:35.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836484 2024-09-18T16:00:35.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836484 2024-09-18T16:00:35.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836484' 2024-09-18T16:00:35.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:00:35.295 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836484 2024-09-18T16:00:35.295 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:00:35.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:00:35.299 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836484 2024-09-18T16:00:35.299 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:00:35.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836484 2024-09-18T16:00:35.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2024-09-18T16:00:35.302 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836484 2024-09-18T16:00:35.303 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:00:35.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836483 -lt 21474836484 2024-09-18T16:00:35.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:00:36.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:00:36.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:00:36.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836483 -lt 21474836484 2024-09-18T16:00:36.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:00:37.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-18T16:00:37.939 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:00:38.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836484 -lt 21474836484 2024-09-18T16:00:38.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:00:38.264 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:00:38.264 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:00:38.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-18T16:00:38.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:00:38.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:00:38.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:00:38.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:00:38.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:00:38.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:00:38.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:00:39.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-18T16:00:39.009 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:00:39.009 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:00:39.009 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:00:39.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-18T16:00:39.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:00:39.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:00:39.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6233: TEST_request_scrub_priority: local osd=0 2024-09-18T16:00:39.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6234: TEST_request_scrub_priority: add_something td/osd-scrub-repair psr_pool POBJ noscrub 2024-09-18T16:00:39.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T16:00:39.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=psr_pool 2024-09-18T16:00:39.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=POBJ 2024-09-18T16:00:39.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T16:00:39.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T16:00:39.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T16:00:39.845 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T16:00:39.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T16:00:40.370 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T16:00:40.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T16:00:40.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T16:00:40.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool psr_pool put POBJ td/osd-scrub-repair/ORIGINAL 2024-09-18T16:00:40.511 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6235: TEST_request_scrub_priority: get_primary psr_pool POBJ 2024-09-18T16:00:40.511 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=psr_pool 2024-09-18T16:00:40.511 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=POBJ 2024-09-18T16:00:40.511 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map psr_pool POBJ 2024-09-18T16:00:40.511 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T16:00:40.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6235: TEST_request_scrub_priority: local primary=0 2024-09-18T16:00:40.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6236: TEST_request_scrub_priority: get_pg psr_pool POBJ 2024-09-18T16:00:40.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=psr_pool 2024-09-18T16:00:40.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=POBJ 2024-09-18T16:00:40.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map psr_pool POBJ 2024-09-18T16:00:40.824 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T16:00:41.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6236: TEST_request_scrub_priority: local pg=1.1 2024-09-18T16:00:41.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6237: TEST_request_scrub_priority: ceph osd dump 2024-09-18T16:00:41.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6237: TEST_request_scrub_priority: grep '^pool.*['\'']psr_pool['\'']' 2024-09-18T16:00:41.177 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6237: TEST_request_scrub_priority: awk '{ print $2 }' 2024-09-18T16:00:41.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6237: TEST_request_scrub_priority: poolid=1 2024-09-18T16:00:41.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6239: TEST_request_scrub_priority: local otherpgs 2024-09-18T16:00:41.485 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6240: TEST_request_scrub_priority: expr 8 - 1 2024-09-18T16:00:41.487 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6240: TEST_request_scrub_priority: seq 0 7 2024-09-18T16:00:41.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6240: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-18T16:00:41.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6242: TEST_request_scrub_priority: opg=1.0 2024-09-18T16:00:41.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: '[' 1.0 = 1.1 ']' 2024-09-18T16:00:41.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: otherpgs='1.0 ' 2024-09-18T16:00:41.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6247: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-18T16:00:41.490 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-09-18T16:00:41.490 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:00:41.490 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:00:41.491 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-18T16:00:41.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6247: TEST_request_scrub_priority: local other_last_scrub=2024-09-18T16:00:33.214857+0000 2024-09-18T16:00:41.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6249: TEST_request_scrub_priority: ceph tell 1.0 scrub 1.0 2024-09-18T16:00:41.949 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:00:41.950 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T16:00:41.950 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T15:59:01.946172+0000" 2024-09-18T16:00:41.950 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:00:41.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6240: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-18T16:00:41.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6242: TEST_request_scrub_priority: opg=1.1 2024-09-18T16:00:41.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: '[' 1.1 = 1.1 ']' 2024-09-18T16:00:41.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6244: TEST_request_scrub_priority: continue 2024-09-18T16:00:41.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6240: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-18T16:00:41.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6242: TEST_request_scrub_priority: opg=1.2 2024-09-18T16:00:41.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: '[' 1.2 = 1.1 ']' 2024-09-18T16:00:41.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: otherpgs='1.0 1.2 ' 2024-09-18T16:00:41.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6247: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-18T16:00:41.962 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-09-18T16:00:41.962 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:00:41.962 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:00:41.962 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-18T16:00:42.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6247: TEST_request_scrub_priority: local other_last_scrub=2024-09-18T16:00:33.214857+0000 2024-09-18T16:00:42.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6249: TEST_request_scrub_priority: ceph tell 1.2 scrub 1.2 2024-09-18T16:00:42.394 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:00:42.394 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T16:00:42.394 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T15:59:02.391087+0000" 2024-09-18T16:00:42.394 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:00:42.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6240: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-18T16:00:42.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6242: TEST_request_scrub_priority: opg=1.3 2024-09-18T16:00:42.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: '[' 1.3 = 1.1 ']' 2024-09-18T16:00:42.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 ' 2024-09-18T16:00:42.406 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6247: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-18T16:00:42.406 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-09-18T16:00:42.406 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:00:42.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:00:42.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-18T16:00:42.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6247: TEST_request_scrub_priority: local other_last_scrub=2024-09-18T16:00:33.214857+0000 2024-09-18T16:00:42.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6249: TEST_request_scrub_priority: ceph tell 1.3 scrub 1.3 2024-09-18T16:00:42.841 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:00:42.841 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T16:00:42.841 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T15:59:02.838068+0000" 2024-09-18T16:00:42.841 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:00:42.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6240: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-18T16:00:42.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6242: TEST_request_scrub_priority: opg=1.4 2024-09-18T16:00:42.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: '[' 1.4 = 1.1 ']' 2024-09-18T16:00:42.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 ' 2024-09-18T16:00:42.853 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6247: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-18T16:00:42.853 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-09-18T16:00:42.853 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:00:42.854 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:00:42.854 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-18T16:00:43.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6247: TEST_request_scrub_priority: local other_last_scrub=2024-09-18T16:00:33.214857+0000 2024-09-18T16:00:43.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6249: TEST_request_scrub_priority: ceph tell 1.4 scrub 1.4 2024-09-18T16:00:43.292 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:00:43.292 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T16:00:43.292 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T15:59:03.288479+0000" 2024-09-18T16:00:43.292 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:00:43.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6240: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-18T16:00:43.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6242: TEST_request_scrub_priority: opg=1.5 2024-09-18T16:00:43.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: '[' 1.5 = 1.1 ']' 2024-09-18T16:00:43.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 ' 2024-09-18T16:00:43.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6247: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-18T16:00:43.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-09-18T16:00:43.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:00:43.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:00:43.305 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-18T16:00:43.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6247: TEST_request_scrub_priority: local other_last_scrub=2024-09-18T16:00:33.214857+0000 2024-09-18T16:00:43.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6249: TEST_request_scrub_priority: ceph tell 1.5 scrub 1.5 2024-09-18T16:00:43.738 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:00:43.738 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T16:00:43.738 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T15:59:03.734924+0000" 2024-09-18T16:00:43.738 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:00:43.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6240: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-18T16:00:43.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6242: TEST_request_scrub_priority: opg=1.6 2024-09-18T16:00:43.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: '[' 1.6 = 1.1 ']' 2024-09-18T16:00:43.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 1.6 ' 2024-09-18T16:00:43.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6247: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-18T16:00:43.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-09-18T16:00:43.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:00:43.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:00:43.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-18T16:00:44.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6247: TEST_request_scrub_priority: local other_last_scrub=2024-09-18T16:00:33.214857+0000 2024-09-18T16:00:44.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6249: TEST_request_scrub_priority: ceph tell 1.6 scrub 1.6 2024-09-18T16:00:44.230 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:00:44.230 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T16:00:44.230 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T15:59:04.227179+0000" 2024-09-18T16:00:44.231 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:00:44.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6240: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-18T16:00:44.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6242: TEST_request_scrub_priority: opg=1.7 2024-09-18T16:00:44.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: '[' 1.7 = 1.1 ']' 2024-09-18T16:00:44.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 1.6 1.7 ' 2024-09-18T16:00:44.247 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6247: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-18T16:00:44.247 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-09-18T16:00:44.247 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:00:44.248 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:00:44.248 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-18T16:00:44.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6247: TEST_request_scrub_priority: local other_last_scrub=2024-09-18T16:00:33.214857+0000 2024-09-18T16:00:44.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6249: TEST_request_scrub_priority: ceph tell 1.7 scrub 1.7 2024-09-18T16:00:44.670 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:00:44.671 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T16:00:44.671 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T15:59:04.667522+0000" 2024-09-18T16:00:44.671 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:00:44.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6252: TEST_request_scrub_priority: sleep 15 2024-09-18T16:00:59.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6253: TEST_request_scrub_priority: flush_pg_stats 2024-09-18T16:00:59.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:00:59.684 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:01:00.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T16:01:00.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:01:00.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:01:00.034 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:01:00.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836490 2024-09-18T16:01:00.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836490 2024-09-18T16:01:00.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836490' 2024-09-18T16:01:00.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:01:00.173 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836490 2024-09-18T16:01:00.173 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:01:00.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:01:00.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836490 2024-09-18T16:01:00.177 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:01:00.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836490 2024-09-18T16:01:00.178 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836490 2024-09-18T16:01:00.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-09-18T16:01:00.179 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:01:00.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836489 -lt 21474836490 2024-09-18T16:01:00.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:01:01.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:01:01.494 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:01:01.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836490 -lt 21474836490 2024-09-18T16:01:01.830 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6256: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-18T16:01:01.830 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-09-18T16:01:01.830 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:01:01.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:01:01.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-18T16:01:02.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6256: TEST_request_scrub_priority: local last_scrub=2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:02.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6257: TEST_request_scrub_priority: ceph pg scrub 1.1 2024-09-18T16:01:02.412 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 1.1 on osd.0 to scrub 2024-09-18T16:01:02.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6259: TEST_request_scrub_priority: ceph osd unset noscrub 2024-09-18T16:01:02.857 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is unset 2024-09-18T16:01:02.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6260: TEST_request_scrub_priority: ceph osd unset nodeep-scrub 2024-09-18T16:01:03.468 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is unset 2024-09-18T16:01:03.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6262: TEST_request_scrub_priority: wait_for_scrub 1.1 2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:03.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.1 2024-09-18T16:01:03.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:03.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T16:01:03.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T16:01:03.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:01:03.486 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-09-18T16:01:03.486 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-09-18T16:01:03.486 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:01:03.487 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:01:03.487 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-18T16:01:03.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:00:33.214857+0000 '>' 2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:03.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:01:04.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:01:04.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:01:04.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-09-18T16:01:04.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-09-18T16:01:04.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:01:04.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:01:04.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-18T16:01:05.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:00:33.214857+0000 '>' 2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:05.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:01:06.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:01:06.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:01:06.142 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-09-18T16:01:06.142 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-09-18T16:01:06.142 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:01:06.143 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:01:06.143 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-18T16:01:06.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:00:33.214857+0000 '>' 2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:06.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:01:07.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:01:07.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:01:07.458 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-09-18T16:01:07.458 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-09-18T16:01:07.458 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:01:07.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:01:07.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-18T16:01:07.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:00:33.214857+0000 '>' 2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:07.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:01:08.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:01:08.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:01:08.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-09-18T16:01:08.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-09-18T16:01:08.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:01:08.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:01:08.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-18T16:01:09.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:01:03.095401+0000 '>' 2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:09.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T16:01:09.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6264: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-18T16:01:09.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6266: TEST_request_scrub_priority: wait_for_scrub 1.0 2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:09.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-18T16:01:09.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:09.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T16:01:09.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T16:01:09.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:01:09.095 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:01:09.095 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:01:09.095 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:01:09.095 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:01:09.095 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:01:09.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:01:04.128395+0000 '>' 2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:09.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T16:01:09.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6264: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-18T16:01:09.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6266: TEST_request_scrub_priority: wait_for_scrub 1.2 2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:09.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.2 2024-09-18T16:01:09.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:09.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T16:01:09.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T16:01:09.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:01:09.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.2 last_scrub_stamp 2024-09-18T16:01:09.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-09-18T16:01:09.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:01:09.436 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:01:09.436 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-09-18T16:01:09.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:01:05.163789+0000 '>' 2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:09.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T16:01:09.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6264: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-18T16:01:09.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6266: TEST_request_scrub_priority: wait_for_scrub 1.3 2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:09.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.3 2024-09-18T16:01:09.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:09.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T16:01:09.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T16:01:09.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:01:09.753 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-18T16:01:09.753 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T16:01:09.753 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:01:09.753 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:01:09.754 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T16:01:10.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:01:06.122759+0000 '>' 2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:10.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T16:01:10.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6264: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-18T16:01:10.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6266: TEST_request_scrub_priority: wait_for_scrub 1.4 2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:10.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.4 2024-09-18T16:01:10.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:10.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T16:01:10.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T16:01:10.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:01:10.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.4 last_scrub_stamp 2024-09-18T16:01:10.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-09-18T16:01:10.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:01:10.095 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:01:10.095 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-09-18T16:01:10.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:01:07.107776+0000 '>' 2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:10.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T16:01:10.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6264: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-18T16:01:10.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6266: TEST_request_scrub_priority: wait_for_scrub 1.5 2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:10.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.5 2024-09-18T16:01:10.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:10.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T16:01:10.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T16:01:10.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:01:10.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2024-09-18T16:01:10.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-09-18T16:01:10.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:01:10.399 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:01:10.399 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-09-18T16:01:10.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T15:59:03.734924+0000 '>' 2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:10.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:01:11.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:01:11.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:01:11.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2024-09-18T16:01:11.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-09-18T16:01:11.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:01:11.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:01:11.702 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-09-18T16:01:12.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T15:59:03.734924+0000 '>' 2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:12.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:01:13.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:01:13.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:01:13.020 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2024-09-18T16:01:13.020 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-09-18T16:01:13.020 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:01:13.021 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:01:13.021 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-09-18T16:01:13.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:01:08.149972+0000 '>' 2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:13.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T16:01:13.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6264: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-18T16:01:13.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6266: TEST_request_scrub_priority: wait_for_scrub 1.6 2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:13.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.6 2024-09-18T16:01:13.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:13.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T16:01:13.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T16:01:13.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:01:13.335 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2024-09-18T16:01:13.335 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-09-18T16:01:13.335 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:01:13.335 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:01:13.335 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-09-18T16:01:13.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:01:09.177180+0000 '>' 2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:13.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T16:01:13.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6264: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-18T16:01:13.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6266: TEST_request_scrub_priority: wait_for_scrub 1.7 2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:13.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.7 2024-09-18T16:01:13.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:13.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T16:01:13.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T16:01:13.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:01:13.632 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.7 last_scrub_stamp 2024-09-18T16:01:13.632 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-09-18T16:01:13.633 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:01:13.633 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:01:13.633 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-09-18T16:01:13.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:01:10.199450+0000 '>' 2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:13.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T16:01:13.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6264: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-18T16:01:13.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6266: TEST_request_scrub_priority: wait_for_scrub 1.1 2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:13.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.1 2024-09-18T16:01:13.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:13.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T16:01:13.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T16:01:13.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:01:13.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-09-18T16:01:13.934 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-09-18T16:01:13.934 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:01:13.934 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:01:13.934 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-18T16:01:14.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:01:03.095401+0000 '>' 2024-09-18T16:00:33.214857+0000 2024-09-18T16:01:14.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T16:01:14.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6270: TEST_request_scrub_priority: grep -v purged_snaps 2024-09-18T16:01:14.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6270: TEST_request_scrub_priority: grep 'log_channel.*scrub ok' td/osd-scrub-repair/osd.0.log 2024-09-18T16:01:14.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6270: TEST_request_scrub_priority: head -1 2024-09-18T16:01:14.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6270: TEST_request_scrub_priority: sed 's/.*[[]DBG[]]//' 2024-09-18T16:01:14.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6270: TEST_request_scrub_priority: grep -q 1.1 2024-09-18T16:01:14.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T16:01:14.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T16:01:14.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:01:14.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T16:01:14.234 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:01:14.234 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:01:14.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:01:14.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:01:14.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:01:14.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:01:14.351 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:01:14.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:01:14.354 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:01:14.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:01:14.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:01:14.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:01:14.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:01:14.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:01:14.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:01:14.361 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:01:14.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:01:14.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:01:14.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T16:01:14.378 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:01:14.378 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:01:14.378 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:01:14.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T16:01:14.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:01:14.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:01:14.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T16:01:14.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T16:01:14.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T16:01:14.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T16:01:14.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T16:01:14.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:01:14.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T16:01:14.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:01:14.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:01:14.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:01:14.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:01:14.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:01:14.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:01:14.390 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:01:14.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:01:14.393 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:01:14.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:01:14.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:01:14.396 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:01:14.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:01:14.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:01:14.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:01:14.400 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:01:14.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:01:14.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:01:14.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T16:01:14.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:01:14.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:01:14.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:01:14.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T16:01:14.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:01:14.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:01:14.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T16:01:14.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T16:01:14.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:01:14.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:01:14.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T16:01:14.417 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T16:01:14.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T16:01:14.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T16:01:14.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T16:01:14.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_scrub_warning td/osd-scrub-repair 2024-09-18T16:01:14.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: local dir=td/osd-scrub-repair 2024-09-18T16:01:14.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: local poolname=psr_pool 2024-09-18T16:01:14.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: local objname=POBJ 2024-09-18T16:01:14.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: local scrubs=5 2024-09-18T16:01:14.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5867: TEST_scrub_warning: local deep_scrubs=5 2024-09-18T16:01:14.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5868: TEST_scrub_warning: local i1_day=86400 2024-09-18T16:01:14.419 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: calc 86400 '*' 7 2024-09-18T16:01:14.419 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: calc: awk 'BEGIN{print 86400 * 7}' 2024-09-18T16:01:14.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: local i7_days=604800 2024-09-18T16:01:14.424 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5870: TEST_scrub_warning: calc 86400 '*' 14 2024-09-18T16:01:14.424 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: calc: awk 'BEGIN{print 86400 * 14}' 2024-09-18T16:01:14.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5870: TEST_scrub_warning: local i14_days=1209600 2024-09-18T16:01:14.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5871: TEST_scrub_warning: local overdue=0.5 2024-09-18T16:01:14.429 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5872: TEST_scrub_warning: calc 604800 + 86400 + '(' 604800 '*' 0.5 ')' 2024-09-18T16:01:14.429 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: calc: awk 'BEGIN{print 604800 + 86400 + ( 604800 * 0.5 )}' 2024-09-18T16:01:14.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5872: TEST_scrub_warning: local conf_overdue_seconds=993600 2024-09-18T16:01:14.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5873: TEST_scrub_warning: calc 1209600 + 86400 + '(' 1209600 '*' 0.5 ')' 2024-09-18T16:01:14.436 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: calc: awk 'BEGIN{print 1209600 + 86400 + ( 1209600 * 0.5 )}' 2024-09-18T16:01:14.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5873: TEST_scrub_warning: local pool_overdue_seconds=1900800 2024-09-18T16:01:14.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5875: TEST_scrub_warning: run_mon td/osd-scrub-repair a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T16:01:14.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T16:01:14.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T16:01:14.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T16:01:14.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T16:01:14.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T16:01:14.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T16:01:14.815 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T16:01:14.815 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:01:14.815 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:01:14.816 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:01:14.816 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:01:14.816 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:01:14.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:01:14.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T16:01:15.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T16:01:15.098 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T16:01:15.098 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:01:15.098 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:01:15.098 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T16:01:15.099 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T16:01:15.100 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:01:15.100 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:01:15.100 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:01:15.101 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:01:15.101 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:01:15.101 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:01:15.101 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T16:01:15.102 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:01:15.103 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T16:01:15.204 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T16:01:15.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:01:15.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:01:15.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T16:01:15.206 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T16:01:15.206 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:01:15.206 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:01:15.206 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:01:15.206 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:01:15.206 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:01:15.207 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:01:15.207 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T16:01:15.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:01:15.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T16:01:15.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5876: TEST_scrub_warning: run_mgr td/osd-scrub-repair x --mon_warn_pg_not_scrubbed_ratio=0.5 --mon_warn_pg_not_deep_scrubbed_ratio=0.5 2024-09-18T16:01:15.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T16:01:15.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T16:01:15.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T16:01:15.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T16:01:15.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T16:01:15.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T16:01:16.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T16:01:16.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:01:16.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:01:16.154 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:01:16.154 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:01:16.154 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:01:16.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:01:16.156 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:01:16.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mon_warn_pg_not_scrubbed_ratio=0.5 --mon_warn_pg_not_deep_scrubbed_ratio=0.5 2024-09-18T16:01:16.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5877: TEST_scrub_warning: run_osd td/osd-scrub-repair 0 --osd_scrub_backoff_ratio=0 2024-09-18T16:01:16.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T16:01:16.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:01:16.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T16:01:16.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:01:16.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T16:01:16.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T16:01:16.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:01:16.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:01:16.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:01:16.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T16:01:16.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T16:01:16.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:01:16.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:01:16.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T16:01:16.195 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:01:16.195 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:01:16.195 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:01:16.195 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:01:16.196 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:01:16.196 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:01:16.196 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:01:16.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:01:16.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:01:16.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:01:16.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:01:16.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T16:01:16.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T16:01:16.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:01:16.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:01:16.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:01:16.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:01:16.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:01:16.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_backoff_ratio=0 2024-09-18T16:01:16.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T16:01:16.200 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:01:16.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6833cb02-32ae-45d5-8629-bc4f0cc54e5a 2024-09-18T16:01:16.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6833cb02-32ae-45d5-8629-bc4f0cc54e5a' 2024-09-18T16:01:16.202 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 6833cb02-32ae-45d5-8629-bc4f0cc54e5a 2024-09-18T16:01:16.202 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:01:16.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBM+epmnSjvDBAALs/YiSx5joehRk0SXPc6jg== 2024-09-18T16:01:16.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBM+epmnSjvDBAALs/YiSx5joehRk0SXPc6jg=="}' 2024-09-18T16:01:16.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6833cb02-32ae-45d5-8629-bc4f0cc54e5a -i td/osd-scrub-repair/0/new.json 2024-09-18T16:01:16.522 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:01:16.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-18T16:01:16.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_backoff_ratio=0 --mkfs --key AQBM+epmnSjvDBAALs/YiSx5joehRk0SXPc6jg== --osd-uuid 6833cb02-32ae-45d5-8629-bc4f0cc54e5a 2024-09-18T16:01:16.567 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:01:16.562+0000 7ff765d97540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:01:16.570 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:01:16.562+0000 7ff765d97540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:01:16.572 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:01:16.566+0000 7ff765d97540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:01:16.572 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:01:16.566+0000 7ff765d97540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T16:01:16.572 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:01:16.566+0000 7ff765d97540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T16:01:16.573 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:01:16.566+0000 7ff765d97540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T16:01:16.646 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:01:16.638+0000 7ff765d97540 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-18T16:01:22.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T16:01:22.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:01:22.826 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T16:01:22.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T16:01:22.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:01:23.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T16:01:23.211 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T16:01:23.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:01:23.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_backoff_ratio=0 2024-09-18T16:01:23.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:01:23.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:01:23.242 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:01:23.234+0000 7f319d316540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:01:23.244 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:01:23.238+0000 7f319d316540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:01:23.247 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:01:23.238+0000 7f319d316540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:01:23.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T16:01:23.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:01:23.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:01:23.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:01:23.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:01:23.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:01:23.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:01:23.561 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:01:23.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:01:23.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:01:23.785 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:01:23.778+0000 7f319d316540 -1 Falling back to public interface 2024-09-18T16:01:23.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:01:24.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:01:24.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:01:24.864 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:01:24.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:01:24.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:01:24.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:01:25.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:01:26.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:01:26.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:01:26.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:01:26.183 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:01:26.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:01:26.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:01:26.459 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:01:26.454+0000 7f319d316540 -1 osd.0 0 log_to_monitors true 2024-09-18T16:01:26.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:01:27.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:01:27.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:01:27.499 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:01:27.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:01:27.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:01:27.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:01:27.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:01:28.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:01:28.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:01:28.808 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:01:28.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:01:28.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:01:28.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:01:29.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:01:30.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:01:30.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:01:30.177 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:01:30.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:01:30.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:01:30.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:01:30.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:01:31.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:01:31.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:01:31.494 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:01:31.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:01:31.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:01:31.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:01:31.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:01:32.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:01:32.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:01:32.813 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:01:32.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:01:32.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:01:32.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:01:33.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:01:34.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:01:34.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:01:34.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:01:34.267 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:01:34.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:01:34.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:01:34.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:01:35.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:01:35.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:01:35.577 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:01:35.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:01:35.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:01:35.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:01:35.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:01:36.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:01:36.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:01:36.889 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:01:36.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:01:36.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:01:36.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:01:37.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:01:38.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:01:38.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:01:38.262 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:01:38.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:01:38.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:01:38.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:01:38.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:01:39.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:01:39.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:01:39.577 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:01:39.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:01:39.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:01:39.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:01:39.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:01:40.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:01:40.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:01:40.951 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:01:40.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:01:40.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:01:40.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:01:41.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:01:42.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:01:42.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:01:42.505 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:01:42.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:01:42.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:01:42.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:01:43.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:01:44.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:01:44.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:01:44.087 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:01:44.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:01:44.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:01:44.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:01:44.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:01:45.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:01:45.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:01:45.730 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:01:45.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:01:45.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:01:45.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:01:46.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:01:47.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:01:47.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:01:47.261 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:01:47.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:01:47.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:01:47.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:01:48.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:01:49.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:01:49.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:01:49.039 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:01:49.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:01:49.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:01:49.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:01:49.352 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/468234,v1:127.0.0.1:6803/468234] [v2:127.0.0.1:6804/468234,v1:127.0.0.1:6805/468234] exists,up 6833cb02-32ae-45d5-8629-bc4f0cc54e5a 2024-09-18T16:01:49.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:01:49.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:01:49.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:01:49.354 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: expr 5 + 5 2024-09-18T16:01:49.356 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: seq 1 10 2024-09-18T16:01:49.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-18T16:01:49.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5881: TEST_scrub_warning: create_pool psr_pool-1 1 1 2024-09-18T16:01:49.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-1 1 1 2024-09-18T16:01:49.919 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'psr_pool-1' created 2024-09-18T16:01:49.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T16:01:50.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5882: TEST_scrub_warning: wait_for_clean 2024-09-18T16:01:50.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:01:50.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:01:50.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:01:50.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:01:50.937 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:01:50.938 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:01:50.938 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:01:50.939 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:01:50.939 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:01:50.939 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:01:51.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:01:51.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:01:51.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:01:51.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:01:51.189 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:01:51.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T16:01:51.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:01:51.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:01:51.505 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:01:51.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836487 2024-09-18T16:01:51.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836487 2024-09-18T16:01:51.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836487' 2024-09-18T16:01:51.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:01:51.685 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836487 2024-09-18T16:01:51.685 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:01:51.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:01:51.689 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836487 2024-09-18T16:01:51.689 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:01:51.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836487 2024-09-18T16:01:51.691 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836487 2024-09-18T16:01:51.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-09-18T16:01:51.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:01:52.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 6 -lt 21474836487 2024-09-18T16:01:52.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:01:53.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:01:53.005 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:01:53.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 6 -lt 21474836487 2024-09-18T16:01:53.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:01:54.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-18T16:01:54.324 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:01:54.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836487 -lt 21474836487 2024-09-18T16:01:54.645 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:01:54.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:01:54.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:01:55.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:01:55.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:01:55.011 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:01:55.011 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:01:55.011 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:01:55.011 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:01:55.012 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:01:55.012 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:01:55.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:01:55.354 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:01:55.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:01:55.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:01:55.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:01:55.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:01:55.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:01:55.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: '[' 1 = 1 ']' 2024-09-18T16:01:55.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5885: TEST_scrub_warning: ceph osd pool set psr_pool-1 scrub_max_interval 1209600 2024-09-18T16:01:56.194 INFO:tasks.workunit.client.0.smithi046.stderr:set pool 1 scrub_max_interval to 1209600 2024-09-18T16:01:56.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5887: TEST_scrub_warning: expr 5 + 1 2024-09-18T16:01:56.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5887: TEST_scrub_warning: '[' 1 = 6 ']' 2024-09-18T16:01:56.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-18T16:01:56.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5881: TEST_scrub_warning: create_pool psr_pool-2 1 1 2024-09-18T16:01:56.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-2 1 1 2024-09-18T16:01:56.706 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'psr_pool-2' created 2024-09-18T16:01:56.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T16:01:57.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5882: TEST_scrub_warning: wait_for_clean 2024-09-18T16:01:57.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:01:57.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:01:57.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:01:57.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:01:57.723 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:01:57.724 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:01:57.724 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:01:57.725 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:01:57.725 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:01:57.725 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:01:57.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:01:57.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:01:57.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:01:57.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:01:57.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:01:58.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T16:01:58.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:01:58.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:01:58.299 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:01:58.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836489 2024-09-18T16:01:58.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836489 2024-09-18T16:01:58.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836489' 2024-09-18T16:01:58.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:01:58.439 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836489 2024-09-18T16:01:58.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:01:58.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:01:58.444 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836489 2024-09-18T16:01:58.444 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:01:58.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836489 2024-09-18T16:01:58.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-09-18T16:01:58.446 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836489 2024-09-18T16:01:58.446 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:01:58.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836488 -lt 21474836489 2024-09-18T16:01:58.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:01:59.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:01:59.760 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:02:00.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836490 -lt 21474836489 2024-09-18T16:02:00.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:02:00.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:02:00.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:02:00.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-18T16:02:00.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:02:00.485 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:02:00.485 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:02:00.485 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:02:00.485 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:02:00.486 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:02:00.486 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:02:00.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T16:02:00.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:02:00.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:02:00.824 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:02:01.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-18T16:02:01.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:02:01.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:02:01.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: '[' 2 = 1 ']' 2024-09-18T16:02:01.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5887: TEST_scrub_warning: expr 5 + 1 2024-09-18T16:02:01.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5887: TEST_scrub_warning: '[' 2 = 6 ']' 2024-09-18T16:02:01.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-18T16:02:01.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5881: TEST_scrub_warning: create_pool psr_pool-3 1 1 2024-09-18T16:02:01.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-3 1 1 2024-09-18T16:02:01.789 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'psr_pool-3' created 2024-09-18T16:02:01.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T16:02:02.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5882: TEST_scrub_warning: wait_for_clean 2024-09-18T16:02:02.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:02:02.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:02:02.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:02:02.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:02:02.806 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:02:02.806 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:02:02.806 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:02:02.807 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:02:02.808 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:02:02.808 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:02:03.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:02:03.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:02:03.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:02:03.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:02:03.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:02:03.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T16:02:03.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:02:03.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:02:03.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:02:03.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836491 2024-09-18T16:02:03.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836491 2024-09-18T16:02:03.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836491' 2024-09-18T16:02:03.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:02:03.559 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836491 2024-09-18T16:02:03.559 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:02:03.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:02:03.562 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836491 2024-09-18T16:02:03.562 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:02:03.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836491 2024-09-18T16:02:03.565 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836491 2024-09-18T16:02:03.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-09-18T16:02:03.566 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:02:03.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836490 -lt 21474836491 2024-09-18T16:02:03.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:02:04.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:02:04.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:02:05.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836490 -lt 21474836491 2024-09-18T16:02:05.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:02:06.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-18T16:02:06.210 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:02:06.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836492 -lt 21474836491 2024-09-18T16:02:06.576 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:02:06.577 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:02:06.577 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:02:06.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2024-09-18T16:02:06.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:02:06.998 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:02:06.998 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:02:06.998 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:02:06.998 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:02:06.999 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:02:06.999 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:02:07.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-09-18T16:02:07.331 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:02:07.332 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:02:07.332 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:02:07.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2024-09-18T16:02:07.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:02:07.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:02:07.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: '[' 3 = 1 ']' 2024-09-18T16:02:07.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5887: TEST_scrub_warning: expr 5 + 1 2024-09-18T16:02:07.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5887: TEST_scrub_warning: '[' 3 = 6 ']' 2024-09-18T16:02:07.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-18T16:02:07.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5881: TEST_scrub_warning: create_pool psr_pool-4 1 1 2024-09-18T16:02:07.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-4 1 1 2024-09-18T16:02:08.149 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'psr_pool-4' created 2024-09-18T16:02:08.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T16:02:09.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5882: TEST_scrub_warning: wait_for_clean 2024-09-18T16:02:09.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:02:09.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:02:09.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:02:09.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:02:09.165 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:02:09.167 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:02:09.167 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:02:09.167 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:02:09.167 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:02:09.167 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:02:09.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:02:09.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:02:09.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:02:09.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:02:09.417 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:02:09.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T16:02:09.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:02:09.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:02:09.739 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:02:09.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836494 2024-09-18T16:02:09.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836494 2024-09-18T16:02:09.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836494' 2024-09-18T16:02:09.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:02:09.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836494 2024-09-18T16:02:09.916 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:02:09.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:02:09.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836494 2024-09-18T16:02:09.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:02:09.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836494 2024-09-18T16:02:09.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-09-18T16:02:09.923 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836494 2024-09-18T16:02:09.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:02:10.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836493 -lt 21474836494 2024-09-18T16:02:10.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:02:11.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:02:11.242 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:02:11.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836493 -lt 21474836494 2024-09-18T16:02:11.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:02:12.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-18T16:02:12.588 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:02:12.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836494 -lt 21474836494 2024-09-18T16:02:12.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:02:12.916 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:02:12.916 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:02:13.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T16:02:13.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:02:13.327 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:02:13.327 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:02:13.327 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:02:13.327 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:02:13.327 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:02:13.328 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:02:13.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T16:02:13.670 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:02:13.671 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:02:13.671 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:02:14.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T16:02:14.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:02:14.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:02:14.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: '[' 4 = 1 ']' 2024-09-18T16:02:14.039 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5887: TEST_scrub_warning: expr 5 + 1 2024-09-18T16:02:14.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5887: TEST_scrub_warning: '[' 4 = 6 ']' 2024-09-18T16:02:14.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-18T16:02:14.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5881: TEST_scrub_warning: create_pool psr_pool-5 1 1 2024-09-18T16:02:14.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-5 1 1 2024-09-18T16:02:14.515 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'psr_pool-5' created 2024-09-18T16:02:14.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T16:02:15.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5882: TEST_scrub_warning: wait_for_clean 2024-09-18T16:02:15.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:02:15.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:02:15.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:02:15.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:02:15.531 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:02:15.532 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:02:15.532 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:02:15.533 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:02:15.533 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:02:15.533 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:02:15.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:02:15.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:02:15.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:02:15.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:02:15.781 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:02:16.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T16:02:16.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:02:16.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:02:16.104 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:02:16.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836496 2024-09-18T16:02:16.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836496 2024-09-18T16:02:16.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836496' 2024-09-18T16:02:16.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:02:16.293 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836496 2024-09-18T16:02:16.294 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:02:16.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:02:16.298 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836496 2024-09-18T16:02:16.298 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:02:16.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836496 2024-09-18T16:02:16.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-09-18T16:02:16.300 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836496 2024-09-18T16:02:16.301 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:02:16.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836495 -lt 21474836496 2024-09-18T16:02:16.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:02:17.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:02:17.613 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:02:17.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836496 -lt 21474836496 2024-09-18T16:02:17.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:02:17.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:02:17.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:02:18.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T16:02:18.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:02:18.370 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:02:18.370 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:02:18.371 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:02:18.371 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:02:18.371 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:02:18.371 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:02:18.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T16:02:18.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:02:18.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:02:18.714 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:02:19.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T16:02:19.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:02:19.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:02:19.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: '[' 5 = 1 ']' 2024-09-18T16:02:19.083 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5887: TEST_scrub_warning: expr 5 + 1 2024-09-18T16:02:19.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5887: TEST_scrub_warning: '[' 5 = 6 ']' 2024-09-18T16:02:19.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-18T16:02:19.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5881: TEST_scrub_warning: create_pool psr_pool-6 1 1 2024-09-18T16:02:19.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-6 1 1 2024-09-18T16:02:19.566 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'psr_pool-6' created 2024-09-18T16:02:19.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T16:02:20.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5882: TEST_scrub_warning: wait_for_clean 2024-09-18T16:02:20.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:02:20.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:02:20.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:02:20.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:02:20.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:02:20.583 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:02:20.583 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:02:20.584 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:02:20.584 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:02:20.584 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:02:20.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:02:20.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:02:20.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:02:20.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:02:20.820 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:02:21.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T16:02:21.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:02:21.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:02:21.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:02:21.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836498 2024-09-18T16:02:21.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836498 2024-09-18T16:02:21.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498' 2024-09-18T16:02:21.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:02:21.329 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836498 2024-09-18T16:02:21.330 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:02:21.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:02:21.333 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836498 2024-09-18T16:02:21.333 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:02:21.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836498 2024-09-18T16:02:21.335 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836498 2024-09-18T16:02:21.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-09-18T16:02:21.336 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:02:21.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836498 -lt 21474836498 2024-09-18T16:02:21.663 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:02:21.663 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:02:21.664 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:02:22.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-09-18T16:02:22.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:02:22.023 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:02:22.023 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:02:22.023 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:02:22.023 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:02:22.024 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:02:22.024 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:02:22.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-18T16:02:22.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:02:22.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:02:22.359 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:02:22.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-09-18T16:02:22.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:02:22.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:02:22.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: '[' 6 = 1 ']' 2024-09-18T16:02:22.727 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5887: TEST_scrub_warning: expr 5 + 1 2024-09-18T16:02:22.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5887: TEST_scrub_warning: '[' 6 = 6 ']' 2024-09-18T16:02:22.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5889: TEST_scrub_warning: ceph osd pool set psr_pool-6 deep_scrub_interval 1209600 2024-09-18T16:02:23.215 INFO:tasks.workunit.client.0.smithi046.stderr:set pool 6 deep_scrub_interval to 1209600 2024-09-18T16:02:23.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-18T16:02:23.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5881: TEST_scrub_warning: create_pool psr_pool-7 1 1 2024-09-18T16:02:23.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-7 1 1 2024-09-18T16:02:23.693 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'psr_pool-7' created 2024-09-18T16:02:23.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T16:02:24.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5882: TEST_scrub_warning: wait_for_clean 2024-09-18T16:02:24.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:02:24.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:02:24.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:02:24.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:02:24.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:02:24.710 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:02:24.710 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:02:24.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:02:24.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:02:24.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:02:24.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:02:24.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:02:24.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:02:24.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:02:24.957 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:02:25.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T16:02:25.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:02:25.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:02:25.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:02:25.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836500 2024-09-18T16:02:25.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836500 2024-09-18T16:02:25.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500' 2024-09-18T16:02:25.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:02:25.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836500 2024-09-18T16:02:25.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:02:25.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:02:25.417 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836500 2024-09-18T16:02:25.418 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:02:25.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836500 2024-09-18T16:02:25.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-18T16:02:25.420 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836500 2024-09-18T16:02:25.421 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:02:25.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836500 -lt 21474836500 2024-09-18T16:02:25.728 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:02:25.728 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:02:25.729 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:02:26.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 7 == 0 2024-09-18T16:02:26.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:02:26.140 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:02:26.140 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:02:26.140 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:02:26.141 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:02:26.141 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:02:26.141 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:02:26.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=7 2024-09-18T16:02:26.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:02:26.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:02:26.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:02:26.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 7 = 7 2024-09-18T16:02:26.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:02:26.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:02:26.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: '[' 7 = 1 ']' 2024-09-18T16:02:26.849 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5887: TEST_scrub_warning: expr 5 + 1 2024-09-18T16:02:26.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5887: TEST_scrub_warning: '[' 7 = 6 ']' 2024-09-18T16:02:26.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-18T16:02:26.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5881: TEST_scrub_warning: create_pool psr_pool-8 1 1 2024-09-18T16:02:26.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-8 1 1 2024-09-18T16:02:27.328 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'psr_pool-8' created 2024-09-18T16:02:27.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T16:02:28.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5882: TEST_scrub_warning: wait_for_clean 2024-09-18T16:02:28.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:02:28.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:02:28.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:02:28.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:02:28.344 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:02:28.345 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:02:28.345 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:02:28.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:02:28.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:02:28.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:02:28.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:02:28.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:02:28.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:02:28.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:02:28.588 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:02:28.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T16:02:28.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:02:28.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:02:28.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:02:29.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836502 2024-09-18T16:02:29.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836502 2024-09-18T16:02:29.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836502' 2024-09-18T16:02:29.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:02:29.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836502 2024-09-18T16:02:29.083 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:02:29.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:02:29.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836502 2024-09-18T16:02:29.087 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:02:29.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836502 2024-09-18T16:02:29.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-18T16:02:29.089 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836502 2024-09-18T16:02:29.090 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:02:29.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836500 -lt 21474836502 2024-09-18T16:02:29.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:02:30.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:02:30.397 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:02:30.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836502 -lt 21474836502 2024-09-18T16:02:30.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:02:30.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:02:30.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:02:31.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-18T16:02:31.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:02:31.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:02:31.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:02:31.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:02:31.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:02:31.119 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:02:31.119 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:02:31.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-18T16:02:31.457 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:02:31.458 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:02:31.458 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:02:31.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-18T16:02:31.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:02:31.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:02:31.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: '[' 8 = 1 ']' 2024-09-18T16:02:31.848 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5887: TEST_scrub_warning: expr 5 + 1 2024-09-18T16:02:31.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5887: TEST_scrub_warning: '[' 8 = 6 ']' 2024-09-18T16:02:31.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-18T16:02:31.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5881: TEST_scrub_warning: create_pool psr_pool-9 1 1 2024-09-18T16:02:31.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-9 1 1 2024-09-18T16:02:32.326 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'psr_pool-9' created 2024-09-18T16:02:32.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T16:02:33.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5882: TEST_scrub_warning: wait_for_clean 2024-09-18T16:02:33.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:02:33.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:02:33.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:02:33.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:02:33.343 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:02:33.344 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:02:33.345 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:02:33.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:02:33.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:02:33.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:02:33.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:02:33.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:02:33.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:02:33.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:02:33.585 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:02:33.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T16:02:33.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:02:33.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:02:33.939 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:02:34.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836504 2024-09-18T16:02:34.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836504 2024-09-18T16:02:34.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836504' 2024-09-18T16:02:34.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:02:34.117 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836504 2024-09-18T16:02:34.117 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:02:34.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:02:34.120 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836504 2024-09-18T16:02:34.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:02:34.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836504 2024-09-18T16:02:34.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-18T16:02:34.124 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836504 2024-09-18T16:02:34.125 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:02:34.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836502 -lt 21474836504 2024-09-18T16:02:34.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:02:35.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:02:35.447 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:02:35.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836504 -lt 21474836504 2024-09-18T16:02:35.779 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:02:35.780 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:02:35.780 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:02:36.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-18T16:02:36.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:02:36.192 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:02:36.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:02:36.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:02:36.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:02:36.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:02:36.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:02:36.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-18T16:02:36.538 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:02:36.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:02:36.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:02:36.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-18T16:02:36.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:02:36.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:02:36.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: '[' 9 = 1 ']' 2024-09-18T16:02:36.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5887: TEST_scrub_warning: expr 5 + 1 2024-09-18T16:02:36.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5887: TEST_scrub_warning: '[' 9 = 6 ']' 2024-09-18T16:02:36.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-18T16:02:36.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5881: TEST_scrub_warning: create_pool psr_pool-10 1 1 2024-09-18T16:02:36.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-10 1 1 2024-09-18T16:02:37.460 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'psr_pool-10' created 2024-09-18T16:02:37.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T16:02:38.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5882: TEST_scrub_warning: wait_for_clean 2024-09-18T16:02:38.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:02:38.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:02:38.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:02:38.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:02:38.478 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:02:38.479 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:02:38.479 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:02:38.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:02:38.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:02:38.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:02:38.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:02:38.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:02:38.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:02:38.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:02:38.724 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:02:39.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T16:02:39.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:02:39.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:02:39.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:02:39.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836506 2024-09-18T16:02:39.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836506 2024-09-18T16:02:39.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836506' 2024-09-18T16:02:39.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:02:39.235 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836506 2024-09-18T16:02:39.235 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:02:39.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:02:39.238 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836506 2024-09-18T16:02:39.238 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:02:39.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836506 2024-09-18T16:02:39.240 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836506 2024-09-18T16:02:39.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-18T16:02:39.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:02:39.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836504 -lt 21474836506 2024-09-18T16:02:39.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:02:40.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:02:40.555 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:02:40.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836506 -lt 21474836506 2024-09-18T16:02:40.887 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:02:40.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:02:40.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:02:41.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 10 == 0 2024-09-18T16:02:41.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:02:41.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:02:41.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:02:41.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:02:41.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:02:41.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:02:41.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:02:41.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=10 2024-09-18T16:02:41.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:02:41.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:02:41.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:02:41.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 10 = 10 2024-09-18T16:02:41.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:02:41.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:02:41.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: '[' 10 = 1 ']' 2024-09-18T16:02:41.967 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5887: TEST_scrub_warning: expr 5 + 1 2024-09-18T16:02:41.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5887: TEST_scrub_warning: '[' 10 = 6 ']' 2024-09-18T16:02:41.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: local primary=0 2024-09-18T16:02:41.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: ceph osd set noscrub 2024-09-18T16:02:42.451 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T16:02:42.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5897: TEST_scrub_warning: ceph osd set nodeep-scrub 2024-09-18T16:02:43.021 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T16:02:43.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5898: TEST_scrub_warning: ceph config set global osd_scrub_interval_randomize_ratio 0 2024-09-18T16:02:43.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: ceph config set global osd_deep_scrub_randomize_ratio 0 2024-09-18T16:02:43.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5900: TEST_scrub_warning: ceph config set global osd_scrub_max_interval 604800 2024-09-18T16:02:44.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5901: TEST_scrub_warning: ceph config set global osd_deep_scrub_interval 604800 2024-09-18T16:02:44.802 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5904: TEST_scrub_warning: seq 1 5 2024-09-18T16:02:44.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5904: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-09-18T16:02:44.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5906: TEST_scrub_warning: '[' 1 = 1 ']' 2024-09-18T16:02:44.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5908: TEST_scrub_warning: overdue_seconds=1900800 2024-09-18T16:02:44.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5912: TEST_scrub_warning: expr 1900800 + 100 2024-09-18T16:02:44.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5912: TEST_scrub_warning: ceph tell 1.0 scrub 1900900 2024-09-18T16:02:44.943 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:02:44.943 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T16:02:44.943 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-08-27T15:59:24.939499+0000" 2024-09-18T16:02:44.943 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:02:44.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5904: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-09-18T16:02:44.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5906: TEST_scrub_warning: '[' 2 = 1 ']' 2024-09-18T16:02:44.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5910: TEST_scrub_warning: overdue_seconds=993600 2024-09-18T16:02:44.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5912: TEST_scrub_warning: expr 993600 + 200 2024-09-18T16:02:44.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5912: TEST_scrub_warning: ceph tell 2.0 scrub 993800 2024-09-18T16:02:45.085 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:02:45.085 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T16:02:45.085 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-07T03:57:45.081728+0000" 2024-09-18T16:02:45.085 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:02:45.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5904: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-09-18T16:02:45.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5906: TEST_scrub_warning: '[' 3 = 1 ']' 2024-09-18T16:02:45.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5910: TEST_scrub_warning: overdue_seconds=993600 2024-09-18T16:02:45.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5912: TEST_scrub_warning: expr 993600 + 300 2024-09-18T16:02:45.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5912: TEST_scrub_warning: ceph tell 3.0 scrub 993900 2024-09-18T16:02:45.227 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:02:45.227 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T16:02:45.227 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-07T03:56:05.223713+0000" 2024-09-18T16:02:45.227 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:02:45.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5904: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-09-18T16:02:45.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5906: TEST_scrub_warning: '[' 4 = 1 ']' 2024-09-18T16:02:45.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5910: TEST_scrub_warning: overdue_seconds=993600 2024-09-18T16:02:45.238 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5912: TEST_scrub_warning: expr 993600 + 400 2024-09-18T16:02:45.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5912: TEST_scrub_warning: ceph tell 4.0 scrub 994000 2024-09-18T16:02:45.374 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:02:45.374 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T16:02:45.374 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-07T03:54:25.370430+0000" 2024-09-18T16:02:45.374 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:02:45.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5904: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-09-18T16:02:45.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5906: TEST_scrub_warning: '[' 5 = 1 ']' 2024-09-18T16:02:45.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5910: TEST_scrub_warning: overdue_seconds=993600 2024-09-18T16:02:45.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5912: TEST_scrub_warning: expr 993600 + 500 2024-09-18T16:02:45.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5912: TEST_scrub_warning: ceph tell 5.0 scrub 994100 2024-09-18T16:02:45.523 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:02:45.523 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T16:02:45.523 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-07T03:52:45.519534+0000" 2024-09-18T16:02:45.523 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:02:45.535 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5915: TEST_scrub_warning: expr 5 + 1 2024-09-18T16:02:45.536 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5915: TEST_scrub_warning: expr 5 + 5 2024-09-18T16:02:45.538 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5915: TEST_scrub_warning: seq 6 10 2024-09-18T16:02:45.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5915: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-09-18T16:02:45.540 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_scrub_warning: expr 5 + 1 2024-09-18T16:02:45.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_scrub_warning: '[' 6 = 6 ']' 2024-09-18T16:02:45.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5919: TEST_scrub_warning: overdue_seconds=1900800 2024-09-18T16:02:45.542 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5923: TEST_scrub_warning: expr 1900800 + 600 2024-09-18T16:02:45.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5923: TEST_scrub_warning: ceph tell 6.0 deep_scrub 1901400 2024-09-18T16:02:45.681 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:02:45.681 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": true, 2024-09-18T16:02:45.681 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-08-27T15:51:05.677537+0000" 2024-09-18T16:02:45.681 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:02:45.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5915: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-09-18T16:02:45.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_scrub_warning: expr 5 + 1 2024-09-18T16:02:45.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_scrub_warning: '[' 7 = 6 ']' 2024-09-18T16:02:45.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5921: TEST_scrub_warning: overdue_seconds=993600 2024-09-18T16:02:45.696 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5923: TEST_scrub_warning: expr 993600 + 700 2024-09-18T16:02:45.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5923: TEST_scrub_warning: ceph tell 7.0 deep_scrub 994300 2024-09-18T16:02:45.828 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:02:45.828 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": true, 2024-09-18T16:02:45.828 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-07T03:49:25.824277+0000" 2024-09-18T16:02:45.828 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:02:45.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5915: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-09-18T16:02:45.844 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_scrub_warning: expr 5 + 1 2024-09-18T16:02:45.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_scrub_warning: '[' 8 = 6 ']' 2024-09-18T16:02:45.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5921: TEST_scrub_warning: overdue_seconds=993600 2024-09-18T16:02:45.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5923: TEST_scrub_warning: expr 993600 + 800 2024-09-18T16:02:45.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5923: TEST_scrub_warning: ceph tell 8.0 deep_scrub 994400 2024-09-18T16:02:45.974 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:02:45.974 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": true, 2024-09-18T16:02:45.974 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-07T03:47:45.970643+0000" 2024-09-18T16:02:45.974 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:02:45.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5915: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-09-18T16:02:45.986 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_scrub_warning: expr 5 + 1 2024-09-18T16:02:45.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_scrub_warning: '[' 9 = 6 ']' 2024-09-18T16:02:45.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5921: TEST_scrub_warning: overdue_seconds=993600 2024-09-18T16:02:45.989 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5923: TEST_scrub_warning: expr 993600 + 900 2024-09-18T16:02:45.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5923: TEST_scrub_warning: ceph tell 9.0 deep_scrub 994500 2024-09-18T16:02:46.123 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:02:46.123 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": true, 2024-09-18T16:02:46.123 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-07T03:46:06.119861+0000" 2024-09-18T16:02:46.123 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:02:46.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5915: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-09-18T16:02:46.136 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_scrub_warning: expr 5 + 1 2024-09-18T16:02:46.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_scrub_warning: '[' 10 = 6 ']' 2024-09-18T16:02:46.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5921: TEST_scrub_warning: overdue_seconds=993600 2024-09-18T16:02:46.138 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5923: TEST_scrub_warning: expr 993600 + 1000 2024-09-18T16:02:46.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5923: TEST_scrub_warning: ceph tell 10.0 deep_scrub 994600 2024-09-18T16:02:46.265 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:02:46.265 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": true, 2024-09-18T16:02:46.265 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-07T03:44:26.262007+0000" 2024-09-18T16:02:46.265 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:02:46.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5925: TEST_scrub_warning: flush_pg_stats 2024-09-18T16:02:46.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:02:46.277 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:02:46.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T16:02:46.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:02:46.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:02:46.587 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:02:46.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836508 2024-09-18T16:02:46.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836508 2024-09-18T16:02:46.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836508' 2024-09-18T16:02:46.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:02:46.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836508 2024-09-18T16:02:46.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:02:46.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:02:46.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836508 2024-09-18T16:02:46.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:02:46.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836508 2024-09-18T16:02:46.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-09-18T16:02:46.767 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836508 2024-09-18T16:02:46.768 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:02:47.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836507 -lt 21474836508 2024-09-18T16:02:47.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:02:48.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:02:48.084 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:02:48.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836508 -lt 21474836508 2024-09-18T16:02:48.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5927: TEST_scrub_warning: ceph health 2024-09-18T16:02:48.775 INFO:tasks.workunit.client.0.smithi046.stdout:HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 5 pgs not deep-scrubbed in time; 10 pgs not scrubbed in time; 10 pool(s) do not have an application enabled; 10 pool(s) have no replicas configured; OSD count 1 < osd_pool_default_size 3 2024-09-18T16:02:48.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5928: TEST_scrub_warning: ceph health detail 2024-09-18T16:02:49.186 INFO:tasks.workunit.client.0.smithi046.stdout:HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 5 pgs not deep-scrubbed in time; 10 pgs not scrubbed in time; 10 pool(s) do not have an application enabled; 10 pool(s) have no replicas configured; OSD count 1 < osd_pool_default_size 3 2024-09-18T16:02:49.186 INFO:tasks.workunit.client.0.smithi046.stdout:[WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-18T16:02:49.186 INFO:tasks.workunit.client.0.smithi046.stdout:[WRN] PG_NOT_DEEP_SCRUBBED: 5 pgs not deep-scrubbed in time 2024-09-18T16:02:49.186 INFO:tasks.workunit.client.0.smithi046.stdout: pg 10.0 not deep-scrubbed since 2024-09-07T03:44:26.262007+0000 2024-09-18T16:02:49.186 INFO:tasks.workunit.client.0.smithi046.stdout: pg 9.0 not deep-scrubbed since 2024-09-07T03:46:06.119861+0000 2024-09-18T16:02:49.186 INFO:tasks.workunit.client.0.smithi046.stdout: pg 8.0 not deep-scrubbed since 2024-09-07T03:47:45.970643+0000 2024-09-18T16:02:49.186 INFO:tasks.workunit.client.0.smithi046.stdout: pg 7.0 not deep-scrubbed since 2024-09-07T03:49:25.824277+0000 2024-09-18T16:02:49.186 INFO:tasks.workunit.client.0.smithi046.stdout: pg 6.0 not deep-scrubbed since 2024-08-27T15:51:05.677537+0000 2024-09-18T16:02:49.186 INFO:tasks.workunit.client.0.smithi046.stdout:[WRN] PG_NOT_SCRUBBED: 10 pgs not scrubbed in time 2024-09-18T16:02:49.187 INFO:tasks.workunit.client.0.smithi046.stdout: pg 10.0 not scrubbed since 2024-09-07T03:44:26.262007+0000 2024-09-18T16:02:49.187 INFO:tasks.workunit.client.0.smithi046.stdout: pg 9.0 not scrubbed since 2024-09-07T03:46:06.119861+0000 2024-09-18T16:02:49.187 INFO:tasks.workunit.client.0.smithi046.stdout: pg 8.0 not scrubbed since 2024-09-07T03:47:45.970643+0000 2024-09-18T16:02:49.187 INFO:tasks.workunit.client.0.smithi046.stdout: pg 7.0 not scrubbed since 2024-09-07T03:49:25.824277+0000 2024-09-18T16:02:49.187 INFO:tasks.workunit.client.0.smithi046.stdout: pg 6.0 not scrubbed since 2024-08-27T15:51:05.677537+0000 2024-09-18T16:02:49.187 INFO:tasks.workunit.client.0.smithi046.stdout: pg 5.0 not scrubbed since 2024-09-07T03:52:45.519534+0000 2024-09-18T16:02:49.187 INFO:tasks.workunit.client.0.smithi046.stdout: pg 4.0 not scrubbed since 2024-09-07T03:54:25.370430+0000 2024-09-18T16:02:49.187 INFO:tasks.workunit.client.0.smithi046.stdout: pg 3.0 not scrubbed since 2024-09-07T03:56:05.223713+0000 2024-09-18T16:02:49.187 INFO:tasks.workunit.client.0.smithi046.stdout: pg 2.0 not scrubbed since 2024-09-07T03:57:45.081728+0000 2024-09-18T16:02:49.187 INFO:tasks.workunit.client.0.smithi046.stdout: pg 1.0 not scrubbed since 2024-08-27T15:59:24.939499+0000 2024-09-18T16:02:49.187 INFO:tasks.workunit.client.0.smithi046.stdout:[WRN] POOL_APP_NOT_ENABLED: 10 pool(s) do not have an application enabled 2024-09-18T16:02:49.187 INFO:tasks.workunit.client.0.smithi046.stdout: application not enabled on pool 'psr_pool-1' 2024-09-18T16:02:49.187 INFO:tasks.workunit.client.0.smithi046.stdout: application not enabled on pool 'psr_pool-2' 2024-09-18T16:02:49.187 INFO:tasks.workunit.client.0.smithi046.stdout: application not enabled on pool 'psr_pool-3' 2024-09-18T16:02:49.188 INFO:tasks.workunit.client.0.smithi046.stdout: application not enabled on pool 'psr_pool-4' 2024-09-18T16:02:49.188 INFO:tasks.workunit.client.0.smithi046.stdout: application not enabled on pool 'psr_pool-5' 2024-09-18T16:02:49.188 INFO:tasks.workunit.client.0.smithi046.stdout: application not enabled on pool 'psr_pool-6' 2024-09-18T16:02:49.188 INFO:tasks.workunit.client.0.smithi046.stdout: application not enabled on pool 'psr_pool-7' 2024-09-18T16:02:49.188 INFO:tasks.workunit.client.0.smithi046.stdout: application not enabled on pool 'psr_pool-8' 2024-09-18T16:02:49.188 INFO:tasks.workunit.client.0.smithi046.stdout: application not enabled on pool 'psr_pool-9' 2024-09-18T16:02:49.188 INFO:tasks.workunit.client.0.smithi046.stdout: application not enabled on pool 'psr_pool-10' 2024-09-18T16:02:49.188 INFO:tasks.workunit.client.0.smithi046.stdout: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-18T16:02:49.188 INFO:tasks.workunit.client.0.smithi046.stdout:[WRN] POOL_NO_REDUNDANCY: 10 pool(s) have no replicas configured 2024-09-18T16:02:49.188 INFO:tasks.workunit.client.0.smithi046.stdout: pool 'psr_pool-1' has no replicas configured 2024-09-18T16:02:49.188 INFO:tasks.workunit.client.0.smithi046.stdout: pool 'psr_pool-2' has no replicas configured 2024-09-18T16:02:49.188 INFO:tasks.workunit.client.0.smithi046.stdout: pool 'psr_pool-3' has no replicas configured 2024-09-18T16:02:49.188 INFO:tasks.workunit.client.0.smithi046.stdout: pool 'psr_pool-4' has no replicas configured 2024-09-18T16:02:49.188 INFO:tasks.workunit.client.0.smithi046.stdout: pool 'psr_pool-5' has no replicas configured 2024-09-18T16:02:49.189 INFO:tasks.workunit.client.0.smithi046.stdout: pool 'psr_pool-6' has no replicas configured 2024-09-18T16:02:49.189 INFO:tasks.workunit.client.0.smithi046.stdout: pool 'psr_pool-7' has no replicas configured 2024-09-18T16:02:49.189 INFO:tasks.workunit.client.0.smithi046.stdout: pool 'psr_pool-8' has no replicas configured 2024-09-18T16:02:49.189 INFO:tasks.workunit.client.0.smithi046.stdout: pool 'psr_pool-9' has no replicas configured 2024-09-18T16:02:49.189 INFO:tasks.workunit.client.0.smithi046.stdout: pool 'psr_pool-10' has no replicas configured 2024-09-18T16:02:49.189 INFO:tasks.workunit.client.0.smithi046.stdout:[WRN] TOO_FEW_OSDS: OSD count 1 < osd_pool_default_size 3 2024-09-18T16:02:49.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5929: TEST_scrub_warning: ceph health 2024-09-18T16:02:49.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5929: TEST_scrub_warning: grep -q ' pgs not deep-scrubbed in time' 2024-09-18T16:02:49.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5930: TEST_scrub_warning: grep -q ' pgs not scrubbed in time' 2024-09-18T16:02:49.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5930: TEST_scrub_warning: ceph health 2024-09-18T16:02:49.978 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5935: TEST_scrub_warning: ceph health detail 2024-09-18T16:02:49.978 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5935: TEST_scrub_warning: grep 'not scrubbed since' 2024-09-18T16:02:49.979 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5935: TEST_scrub_warning: wc -l 2024-09-18T16:02:50.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5935: TEST_scrub_warning: COUNT=10 2024-09-18T16:02:50.395 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5937: TEST_scrub_warning: expr 5+5 2024-09-18T16:02:50.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5937: TEST_scrub_warning: (( 10 != 5 && 10 != 5+5 )) 2024-09-18T16:02:50.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5941: TEST_scrub_warning: ceph health detail 2024-09-18T16:02:50.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5941: TEST_scrub_warning: grep 'not deep-scrubbed since' 2024-09-18T16:02:50.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5941: TEST_scrub_warning: wc -l 2024-09-18T16:02:50.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5941: TEST_scrub_warning: COUNT=5 2024-09-18T16:02:50.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5942: TEST_scrub_warning: '[' 5 '!=' 5 ']' 2024-09-18T16:02:50.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T16:02:50.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T16:02:50.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:02:50.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T16:02:50.788 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:02:50.788 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:02:50.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:02:50.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:02:50.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:02:50.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:02:50.905 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:02:50.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:02:50.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:02:50.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:02:50.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:02:50.912 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:02:50.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:02:50.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:02:50.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:02:50.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:02:50.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:02:50.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:02:50.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T16:02:50.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:02:50.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:02:50.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:02:50.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T16:02:50.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:02:50.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:02:50.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T16:02:50.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T16:02:50.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T16:02:50.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T16:02:50.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T16:02:50.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:02:50.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T16:02:50.937 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:02:50.937 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:02:50.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:02:50.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:02:50.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:02:50.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:02:50.945 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:02:50.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:02:50.948 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:02:50.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:02:50.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:02:50.950 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:02:50.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:02:50.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:02:50.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:02:50.954 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:02:50.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:02:50.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:02:50.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T16:02:50.960 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:02:50.960 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:02:50.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:02:50.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T16:02:50.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:02:50.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:02:50.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T16:02:50.966 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T16:02:50.966 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:02:50.966 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:02:50.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T16:02:50.969 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T16:02:50.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T16:02:50.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T16:02:50.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T16:02:50.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_skip_non_repair_during_recovery td/osd-scrub-repair 2024-09-18T16:02:50.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:135: TEST_skip_non_repair_during_recovery: local dir=td/osd-scrub-repair 2024-09-18T16:02:50.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:136: TEST_skip_non_repair_during_recovery: local poolname=rbd 2024-09-18T16:02:50.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:138: TEST_skip_non_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-09-18T16:02:50.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T16:02:50.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T16:02:50.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T16:02:50.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T16:02:50.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T16:02:50.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-09-18T16:02:51.344 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T16:02:51.344 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:02:51.344 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:02:51.344 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:02:51.344 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:02:51.345 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:02:51.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:02:51.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-18T16:02:51.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T16:02:51.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T16:02:51.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:02:51.637 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:02:51.637 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T16:02:51.637 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T16:02:51.638 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:02:51.638 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:02:51.638 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:02:51.639 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:02:51.639 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:02:51.639 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:02:51.639 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T16:02:51.640 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:02:51.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T16:02:51.735 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T16:02:51.735 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:02:51.735 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:02:51.735 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T16:02:51.736 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:02:51.736 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T16:02:51.736 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:02:51.736 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:02:51.737 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:02:51.737 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:02:51.737 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:02:51.737 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T16:02:51.737 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:02:51.737 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T16:02:51.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:139: TEST_skip_non_repair_during_recovery: run_mgr td/osd-scrub-repair x 2024-09-18T16:02:51.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T16:02:51.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T16:02:51.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T16:02:51.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T16:02:51.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T16:02:51.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T16:02:52.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T16:02:52.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:02:52.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:02:52.706 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:02:52.706 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:02:52.706 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:02:52.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:02:52.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:02:52.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:02:52.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:140: TEST_skip_non_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-18T16:02:52.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T16:02:52.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:02:52.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T16:02:52.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:02:52.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T16:02:52.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T16:02:52.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:02:52.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:02:52.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:02:52.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T16:02:52.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T16:02:52.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:02:52.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:02:52.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T16:02:52.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:02:52.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:02:52.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:02:52.750 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:02:52.750 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:02:52.750 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:02:52.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:02:52.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:02:52.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:02:52.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:02:52.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:02:52.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T16:02:52.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T16:02:52.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:02:52.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:02:52.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:02:52.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:02:52.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:02:52.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-09-18T16:02:52.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T16:02:52.753 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:02:52.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6b1688fe-f8a3-4f63-9941-6ee1e49fdd27 2024-09-18T16:02:52.754 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 6b1688fe-f8a3-4f63-9941-6ee1e49fdd27 2024-09-18T16:02:52.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6b1688fe-f8a3-4f63-9941-6ee1e49fdd27' 2024-09-18T16:02:52.755 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:02:52.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCs+epm+hvpLRAAet7zGX+RT7s88U1ZoVyZiA== 2024-09-18T16:02:52.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCs+epm+hvpLRAAet7zGX+RT7s88U1ZoVyZiA=="}' 2024-09-18T16:02:52.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6b1688fe-f8a3-4f63-9941-6ee1e49fdd27 -i td/osd-scrub-repair/0/new.json 2024-09-18T16:02:53.085 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:02:53.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-18T16:02:53.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQCs+epm+hvpLRAAet7zGX+RT7s88U1ZoVyZiA== --osd-uuid 6b1688fe-f8a3-4f63-9941-6ee1e49fdd27 2024-09-18T16:02:53.134 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:02:53.125+0000 7f9876711540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:02:53.138 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:02:53.129+0000 7f9876711540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:02:53.139 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:02:53.133+0000 7f9876711540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:02:53.141 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:02:53.133+0000 7f9876711540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T16:02:53.141 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:02:53.133+0000 7f9876711540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T16:02:53.141 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:02:53.133+0000 7f9876711540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T16:02:53.193 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:02:53.185+0000 7f9876711540 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-18T16:02:59.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T16:02:59.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:02:59.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T16:02:59.363 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T16:02:59.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:02:59.749 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T16:02:59.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T16:02:59.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-18T16:02:59.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:02:59.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:02:59.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:02:59.780 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:02:59.773+0000 7f6069d9b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:02:59.783 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:02:59.773+0000 7f6069d9b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:02:59.784 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:02:59.777+0000 7f6069d9b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:03:00.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T16:03:00.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:03:00.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:03:00.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:03:00.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:03:00.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:00.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:03:00.105 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:03:00.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:00.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:03:00.329 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:03:00.321+0000 7f6069d9b540 -1 Falling back to public interface 2024-09-18T16:03:00.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:01.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:01.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:01.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:03:01.413 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:03:01.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:01.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:03:01.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:02.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:02.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:02.723 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:03:02.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:03:02.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:02.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:03:03.013 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:03:03.005+0000 7f6069d9b540 -1 osd.0 0 log_to_monitors true 2024-09-18T16:03:03.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:04.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:04.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:04.044 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:03:04.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:03:04.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:04.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:03:04.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:05.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:05.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:05.363 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:03:05.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:03:05.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:05.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:03:05.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:06.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:06.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:06.752 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:03:06.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:03:06.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:06.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:03:07.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:08.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:08.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:08.082 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:03:08.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:03:08.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:08.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:03:08.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:09.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:09.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:09.401 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:03:09.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:03:09.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:09.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:03:09.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:10.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:10.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:10.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:03:10.728 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:03:10.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:10.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:03:11.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:12.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:12.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:12.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:03:12.037 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:03:12.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:12.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:03:12.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:13.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:13.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:13.356 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:03:13.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:03:13.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:13.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:03:13.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:14.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:14.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:14.673 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:03:14.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:03:14.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:14.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:03:15.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:16.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:16.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:16.175 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:03:16.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:03:16.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:16.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:03:16.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:17.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:17.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:17.601 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:03:17.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:03:17.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:17.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:03:18.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:19.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:19.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:19.386 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:03:19.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:03:19.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:19.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:03:19.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:20.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:20.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:20.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:03:20.930 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:03:20.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:20.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:03:21.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:22.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:22.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:22.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:03:22.299 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:03:22.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:03:22.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:22.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:23.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:23.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:23.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:03:23.972 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:03:23.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:23.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:03:24.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:25.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:25.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:25.490 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:03:25.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:03:25.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:25.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:03:25.799 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/474062,v1:127.0.0.1:6803/474062] [v2:127.0.0.1:6804/474062,v1:127.0.0.1:6805/474062] exists,up 6b1688fe-f8a3-4f63-9941-6ee1e49fdd27 2024-09-18T16:03:25.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:03:25.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:03:25.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:03:25.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:143: TEST_skip_non_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-18T16:03:25.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T16:03:25.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:03:25.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T16:03:25.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:03:25.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T16:03:25.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T16:03:25.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:03:25.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:03:25.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:03:25.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T16:03:25.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T16:03:25.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:03:25.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:03:25.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T16:03:25.803 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:03:25.803 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:03:25.803 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:03:25.805 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:03:25.805 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:03:25.805 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:03:25.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:03:25.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:03:25.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:03:25.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:03:25.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:03:25.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T16:03:25.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T16:03:25.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:03:25.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:03:25.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:03:25.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:03:25.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:03:25.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-09-18T16:03:25.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T16:03:25.807 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:03:25.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=de7857d5-e8ad-4b32-804d-e7b9a2b7d975 2024-09-18T16:03:25.809 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 de7857d5-e8ad-4b32-804d-e7b9a2b7d975 2024-09-18T16:03:25.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 de7857d5-e8ad-4b32-804d-e7b9a2b7d975' 2024-09-18T16:03:25.810 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:03:25.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDN+epmi9VgMRAAHVH+Rp5ieeJbR8gUDe0gOw== 2024-09-18T16:03:25.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDN+epmi9VgMRAAHVH+Rp5ieeJbR8gUDe0gOw=="}' 2024-09-18T16:03:25.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new de7857d5-e8ad-4b32-804d-e7b9a2b7d975 -i td/osd-scrub-repair/1/new.json 2024-09-18T16:03:26.293 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:03:26.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-18T16:03:26.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQDN+epmi9VgMRAAHVH+Rp5ieeJbR8gUDe0gOw== --osd-uuid de7857d5-e8ad-4b32-804d-e7b9a2b7d975 2024-09-18T16:03:26.351 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:03:26.341+0000 7f7556fa8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:03:26.356 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:03:26.349+0000 7f7556fa8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:03:26.357 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:03:26.349+0000 7f7556fa8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:03:26.358 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:03:26.349+0000 7f7556fa8540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T16:03:26.358 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:03:26.349+0000 7f7556fa8540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T16:03:26.358 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:03:26.349+0000 7f7556fa8540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T16:03:26.436 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:03:26.429+0000 7f7556fa8540 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-18T16:03:32.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-18T16:03:32.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:03:32.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T16:03:32.656 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T16:03:32.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:03:33.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T16:03:33.055 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T16:03:33.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-18T16:03:33.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:03:33.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:03:33.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:03:33.084 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:03:33.077+0000 7f1d93279540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:03:33.087 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:03:33.077+0000 7f1d93279540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:03:33.088 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:03:33.081+0000 7f1d93279540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:03:33.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T16:03:33.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:03:33.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T16:03:33.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:03:33.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:03:33.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:33.402 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:03:33.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:03:33.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:33.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:03:33.626 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:03:33.617+0000 7f1d93279540 -1 Falling back to public interface 2024-09-18T16:03:33.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:34.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:34.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:34.745 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:03:34.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:03:34.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:34.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:03:35.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:36.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:36.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:36.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:03:36.053 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:03:36.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:36.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:03:36.365 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:03:36.357+0000 7f1d93279540 -1 osd.1 0 log_to_monitors true 2024-09-18T16:03:36.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:37.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:37.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:37.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:03:37.409 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:03:37.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:37.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:03:37.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:38.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:38.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:38.725 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:03:38.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:03:38.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:38.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:03:39.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:40.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:40.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:40.059 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:03:40.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:03:40.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:40.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:03:40.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:41.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:41.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:41.417 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:03:41.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:03:41.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:41.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:03:41.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:42.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:42.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:42.737 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:03:42.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:03:42.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:42.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:03:43.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:44.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:44.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:44.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:03:44.118 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:03:44.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:44.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:03:44.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:45.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:45.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:45.485 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:03:45.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:03:45.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:45.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:03:45.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:46.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:46.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:46.806 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:03:46.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:03:46.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:03:46.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:47.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:48.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:48.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:48.117 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:03:48.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:03:48.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:48.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:03:48.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:49.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:49.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:49.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:03:49.431 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:03:49.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:49.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:03:49.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:50.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:50.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:50.796 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:03:50.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:03:50.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:50.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:03:51.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:52.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:52.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:52.264 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:03:52.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:03:52.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:52.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:03:52.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:53.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:53.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:53.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:03:53.909 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:03:53.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:53.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:03:54.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:55.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:55.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:55.688 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:03:55.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:03:55.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:55.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:03:56.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:57.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:57.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:57.254 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:03:57.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:03:57.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:57.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:03:57.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:03:58.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:03:58.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:03:58.604 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:03:58.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:03:58.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:03:58.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:03:59.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:00.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:00.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:00.014 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T16:04:00.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T16:04:00.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:00.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:04:00.348 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/475841,v1:127.0.0.1:6811/475841] [v2:127.0.0.1:6812/475841,v1:127.0.0.1:6813/475841] exists,up de7857d5-e8ad-4b32-804d-e7b9a2b7d975 2024-09-18T16:04:00.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:04:00.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:04:00.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:04:00.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:146: TEST_skip_non_repair_during_recovery: create_rbd_pool 2024-09-18T16:04:00.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T16:04:00.702 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' does not exist 2024-09-18T16:04:00.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T16:04:00.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T16:04:01.282 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' created 2024-09-18T16:04:01.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T16:04:02.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T16:04:04.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:147: TEST_skip_non_repair_during_recovery: wait_for_clean 2024-09-18T16:04:04.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:04:04.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:04:04.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:04:04.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:04:04.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:04:04.062 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:04:04.062 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:04:04.063 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:04:04.063 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:04:04.063 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:04:04.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:04:04.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:04:04.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:04:04.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:04:04.311 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:04:04.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:04:04.630 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T16:04:04.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:04:04.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:04:04.631 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:04:04.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836494 2024-09-18T16:04:04.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836494 2024-09-18T16:04:04.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836494' 2024-09-18T16:04:04.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:04:04.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:04:04.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672965 2024-09-18T16:04:04.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672965 2024-09-18T16:04:04.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836494 1-42949672965' 2024-09-18T16:04:04.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:04:04.989 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836494 2024-09-18T16:04:04.989 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:04:04.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:04:04.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836494 2024-09-18T16:04:04.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:04:04.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836494 2024-09-18T16:04:04.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-09-18T16:04:04.996 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836494 2024-09-18T16:04:04.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:04:05.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836493 -lt 21474836494 2024-09-18T16:04:05.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:04:06.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:04:06.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:04:06.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836495 -lt 21474836494 2024-09-18T16:04:06.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:04:06.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672965 2024-09-18T16:04:06.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:04:06.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:04:06.639 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672965 2024-09-18T16:04:06.640 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:04:06.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672965 2024-09-18T16:04:06.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-18T16:04:06.642 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672965 2024-09-18T16:04:06.642 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:04:06.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672965 -lt 42949672965 2024-09-18T16:04:06.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:04:06.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:04:06.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:04:07.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T16:04:07.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:04:07.350 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:04:07.350 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:04:07.350 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:04:07.350 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:04:07.351 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:04:07.351 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:04:07.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T16:04:07.725 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:04:07.726 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:04:07.726 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:04:08.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T16:04:08.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:04:08.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:04:08.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:149: TEST_skip_non_repair_during_recovery: add_something td/osd-scrub-repair rbd 2024-09-18T16:04:08.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T16:04:08.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=rbd 2024-09-18T16:04:08.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=SOMETHING 2024-09-18T16:04:08.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T16:04:08.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T16:04:08.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T16:04:08.554 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T16:04:08.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T16:04:09.005 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T16:04:09.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T16:04:09.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T16:04:09.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-18T16:04:09.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:150: TEST_skip_non_repair_during_recovery: get_not_primary rbd SOMETHING 2024-09-18T16:04:09.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-09-18T16:04:09.305 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-09-18T16:04:09.305 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-09-18T16:04:09.305 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-09-18T16:04:09.305 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T16:04:09.306 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-09-18T16:04:09.306 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T16:04:09.642 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T16:04:09.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T16:04:09.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-09-18T16:04:09.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:150: TEST_skip_non_repair_during_recovery: scrub_and_not_schedule td/osd-scrub-repair rbd 0 2024-09-18T16:04:09.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:154: scrub_and_not_schedule: local dir=td/osd-scrub-repair 2024-09-18T16:04:09.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:155: scrub_and_not_schedule: local poolname=rbd 2024-09-18T16:04:09.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:156: scrub_and_not_schedule: local osd=0 2024-09-18T16:04:09.995 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:161: scrub_and_not_schedule: get_pg rbd SOMETHING 2024-09-18T16:04:09.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-09-18T16:04:09.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T16:04:09.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-09-18T16:04:09.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T16:04:10.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:161: scrub_and_not_schedule: local pg=1.3 2024-09-18T16:04:10.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:162: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-09-18T16:04:10.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T16:04:10.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:04:10.341 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:04:10.341 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T16:04:10.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:162: scrub_and_not_schedule: local last_scrub=2024-09-18T16:04:01.109845+0000 2024-09-18T16:04:10.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:163: scrub_and_not_schedule: ceph pg scrub 1.3 2024-09-18T16:04:10.938 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 1.3 on osd.1 to scrub 2024-09-18T16:04:10.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:168: scrub_and_not_schedule: (( i=0 )) 2024-09-18T16:04:10.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:168: scrub_and_not_schedule: (( i < 3 )) 2024-09-18T16:04:10.947 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-09-18T16:04:10.947 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T16:04:10.947 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:04:10.948 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:04:10.948 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T16:04:11.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: test 2024-09-18T16:04:01.109845+0000 '>' 2024-09-18T16:04:01.109845+0000 2024-09-18T16:04:11.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:172: scrub_and_not_schedule: sleep 1 2024-09-18T16:04:12.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:168: scrub_and_not_schedule: (( i++ )) 2024-09-18T16:04:12.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:168: scrub_and_not_schedule: (( i < 3 )) 2024-09-18T16:04:12.256 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-09-18T16:04:12.256 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T16:04:12.256 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:04:12.257 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:04:12.257 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T16:04:12.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: test 2024-09-18T16:04:01.109845+0000 '>' 2024-09-18T16:04:01.109845+0000 2024-09-18T16:04:12.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:172: scrub_and_not_schedule: sleep 1 2024-09-18T16:04:13.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:168: scrub_and_not_schedule: (( i++ )) 2024-09-18T16:04:13.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:168: scrub_and_not_schedule: (( i < 3 )) 2024-09-18T16:04:13.572 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-09-18T16:04:13.572 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-09-18T16:04:13.572 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:04:13.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:04:13.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-18T16:04:13.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: test 2024-09-18T16:04:01.109845+0000 '>' 2024-09-18T16:04:01.109845+0000 2024-09-18T16:04:13.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:172: scrub_and_not_schedule: sleep 1 2024-09-18T16:04:14.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:168: scrub_and_not_schedule: (( i++ )) 2024-09-18T16:04:14.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:168: scrub_and_not_schedule: (( i < 3 )) 2024-09-18T16:04:14.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:178: scrub_and_not_schedule: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-18T16:04:14.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T16:04:14.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T16:04:14.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T16:04:14.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T16:04:14.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-18T16:04:14.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T16:04:14.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T16:04:14.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T16:04:14.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T16:04:14.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T16:04:14.892 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:04:14.892 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:04:14.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:04:14.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:04:14.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:04:16.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:04:16.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-18T16:04:16.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T16:04:16.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T16:04:16.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T16:04:16.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T16:04:16.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T16:04:16.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-09-18T16:04:17.659 INFO:tasks.workunit.client.0.smithi046.stdout:_ 2024-09-18T16:04:17.659 INFO:tasks.workunit.client.0.smithi046.stdout:snapset 2024-09-18T16:04:18.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-18T16:04:18.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T16:04:18.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T16:04:18.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T16:04:18.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T16:04:18.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T16:04:18.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T16:04:18.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:04:18.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:04:18.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:04:18.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T16:04:18.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T16:04:18.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T16:04:18.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T16:04:18.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T16:04:18.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T16:04:18.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:04:18.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:04:18.208 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:04:18.208 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:04:18.208 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:04:18.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:04:18.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:04:18.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:04:18.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T16:04:18.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T16:04:18.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:04:18.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:04:18.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:04:18.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:04:18.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T16:04:18.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T16:04:18.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T16:04:18.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T16:04:18.212 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.0 2024-09-18T16:04:18.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:04:18.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T16:04:18.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T16:04:18.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T16:04:18.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T16:04:18.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T16:04:18.243 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:18.236+0000 7f60698a5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:18.248 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:18.240+0000 7f60698a5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:18.253 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:18.240+0000 7f60698a5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:18.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T16:04:18.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:04:18.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:04:18.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:04:18.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:04:18.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:18.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:04:18.526 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T16:04:18.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:18.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:18.790 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:18.780+0000 7f60698a5540 -1 Falling back to public interface 2024-09-18T16:04:18.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:19.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:19.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:19.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:04:19.931 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:04:19.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:19.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:20.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:21.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:21.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:21.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:04:21.257 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T16:04:21.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:21.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:21.487 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:21.480+0000 7f60698a5540 -1 osd.0 21 log_to_monitors true 2024-09-18T16:04:21.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:22.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:22.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:22.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:04:22.613 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T16:04:22.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:22.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:22.966 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 up in weight 1 up_from 25 up_thru 25 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/477882,v1:127.0.0.1:6803/477882] [v2:127.0.0.1:6804/477882,v1:127.0.0.1:6805/477882] exists,up 6b1688fe-f8a3-4f63-9941-6ee1e49fdd27 2024-09-18T16:04:22.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:04:22.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:04:22.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:04:22.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T16:04:22.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:04:22.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:04:22.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:04:22.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:04:22.968 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:04:22.969 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:04:22.969 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:04:22.969 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:04:22.969 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:04:22.970 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:04:23.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:04:23.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:04:23.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:04:23.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:04:23.209 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:04:23.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:04:23.517 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T16:04:23.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:04:23.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:04:23.518 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:04:23.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182402 2024-09-18T16:04:23.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182402 2024-09-18T16:04:23.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182402' 2024-09-18T16:04:23.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:04:23.665 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:04:23.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672969 2024-09-18T16:04:23.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672969 2024-09-18T16:04:23.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182402 1-42949672969' 2024-09-18T16:04:23.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:04:23.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-107374182402 2024-09-18T16:04:23.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:04:23.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:04:23.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-107374182402 2024-09-18T16:04:23.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:04:23.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182402 2024-09-18T16:04:23.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-09-18T16:04:23.820 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 107374182402 2024-09-18T16:04:23.820 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:04:24.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 107374182402 2024-09-18T16:04:24.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:04:25.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:04:25.133 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:04:25.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182402 -lt 107374182402 2024-09-18T16:04:25.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:04:25.456 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672969 2024-09-18T16:04:25.456 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:04:25.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:04:25.460 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672969 2024-09-18T16:04:25.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:04:25.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672969 2024-09-18T16:04:25.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-09-18T16:04:25.464 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 42949672969 2024-09-18T16:04:25.464 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:04:25.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672969 -lt 42949672969 2024-09-18T16:04:25.790 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:04:25.790 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:04:25.791 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:04:26.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T16:04:26.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:04:26.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:04:26.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:04:26.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:04:26.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:04:26.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:04:26.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:04:26.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T16:04:26.517 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:04:26.518 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:04:26.518 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:04:26.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T16:04:26.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:04:26.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:04:26.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:179: scrub_and_not_schedule: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-09-18T16:04:26.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:180: scrub_and_not_schedule: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-18T16:04:26.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T16:04:26.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T16:04:26.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:04:26.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T16:04:26.949 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:04:26.949 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:04:26.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:04:26.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:04:26.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:04:27.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:04:27.067 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:04:27.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:04:27.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:04:27.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:04:27.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:04:27.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:04:27.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:04:27.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:04:27.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:04:27.078 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:04:27.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:04:27.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:04:27.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T16:04:27.101 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:04:27.101 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:04:27.101 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:04:27.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T16:04:27.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:04:27.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:04:27.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T16:04:27.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T16:04:27.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T16:04:27.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T16:04:27.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T16:04:27.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:04:27.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T16:04:27.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:04:27.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:04:27.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:04:27.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:04:27.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:04:27.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:04:27.112 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:04:27.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:04:27.115 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:04:27.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:04:27.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:04:27.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:04:27.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:04:27.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:04:27.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:04:27.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:04:27.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:04:27.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:04:27.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T16:04:27.128 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:04:27.128 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:04:27.128 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:04:27.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T16:04:27.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:04:27.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:04:27.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T16:04:27.133 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T16:04:27.133 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:04:27.133 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:04:27.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T16:04:27.137 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T16:04:27.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T16:04:27.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T16:04:27.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T16:04:27.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_unfound_erasure_coded_appends td/osd-scrub-repair 2024-09-18T16:04:27.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: TEST_unfound_erasure_coded_appends: unfound_erasure_coded td/osd-scrub-repair false 2024-09-18T16:04:27.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:861: unfound_erasure_coded: local dir=td/osd-scrub-repair 2024-09-18T16:04:27.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:862: unfound_erasure_coded: local allow_overwrites=false 2024-09-18T16:04:27.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: unfound_erasure_coded: local poolname=ecpool 2024-09-18T16:04:27.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: unfound_erasure_coded: local payload=ABCDEF 2024-09-18T16:04:27.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:866: unfound_erasure_coded: run_mon td/osd-scrub-repair a 2024-09-18T16:04:27.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T16:04:27.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T16:04:27.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T16:04:27.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T16:04:27.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T16:04:27.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-18T16:04:27.525 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T16:04:27.525 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:04:27.526 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:04:27.526 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:04:27.526 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:04:27.526 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:04:27.527 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:04:27.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T16:04:27.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T16:04:27.825 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T16:04:27.825 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:04:27.825 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:04:27.825 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T16:04:27.826 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T16:04:27.826 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:04:27.827 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:04:27.827 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:04:27.827 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:04:27.827 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:04:27.827 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:04:27.828 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T16:04:27.828 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:04:27.829 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T16:04:27.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T16:04:27.924 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:04:27.924 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:04:27.924 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T16:04:27.925 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T16:04:27.925 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:04:27.925 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:04:27.925 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:04:27.926 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:04:27.926 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:04:27.926 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:04:27.926 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T16:04:27.927 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:04:27.927 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T16:04:28.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: unfound_erasure_coded: run_mgr td/osd-scrub-repair x 2024-09-18T16:04:28.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T16:04:28.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T16:04:28.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T16:04:28.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T16:04:28.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T16:04:28.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T16:04:28.873 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T16:04:28.874 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:04:28.874 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:04:28.874 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:04:28.874 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:04:28.875 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:04:28.875 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:04:28.877 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:04:28.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:04:28.917 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: seq 0 3 2024-09-18T16:04:28.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-18T16:04:28.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:869: unfound_erasure_coded: '[' false = true ']' 2024-09-18T16:04:28.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:872: unfound_erasure_coded: run_osd_filestore td/osd-scrub-repair 0 2024-09-18T16:04:28.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T16:04:28.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T16:04:28.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=0 2024-09-18T16:04:28.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T16:04:28.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/0 2024-09-18T16:04:28.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T16:04:28.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:04:28.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:04:28.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:04:28.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T16:04:28.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T16:04:28.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T16:04:28.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T16:04:28.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T16:04:28.921 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T16:04:28.921 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:04:28.921 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:04:28.921 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:04:28.921 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:04:28.921 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:04:28.921 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:04:28.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:04:28.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T16:04:28.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T16:04:28.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T16:04:28.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T16:04:28.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T16:04:28.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:04:28.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:04:28.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:04:28.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T16:04:28.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T16:04:28.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/0 2024-09-18T16:04:28.927 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T16:04:28.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=11bf0753-351b-4774-8096-59671f506642 2024-09-18T16:04:28.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd1 11bf0753-351b-4774-8096-59671f506642' 2024-09-18T16:04:28.929 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 11bf0753-351b-4774-8096-59671f506642 2024-09-18T16:04:28.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T16:04:28.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQAM+upmVWdgOBAAt0IrKXim9/NLKZuugmQ09g== 2024-09-18T16:04:28.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQAM+upmVWdgOBAAt0IrKXim9/NLKZuugmQ09g=="}' 2024-09-18T16:04:28.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 11bf0753-351b-4774-8096-59671f506642 -i td/osd-scrub-repair/0/new.json 2024-09-18T16:04:29.223 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:04:29.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/0/new.json 2024-09-18T16:04:29.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAM+upmVWdgOBAAt0IrKXim9/NLKZuugmQ09g== --osd-uuid 11bf0753-351b-4774-8096-59671f506642 --osd-objectstore=filestore 2024-09-18T16:04:29.271 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:29.260+0000 7f0770d55540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:29.318 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:29.308+0000 7f0770d55540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:29.319 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:29.312+0000 7f0770d55540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:29.743 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:29.736+0000 7f0770d55540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T16:04:30.197 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:30.188+0000 7f0770d55540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T16:04:30.197 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:30.188+0000 7f0770d55540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T16:04:30.198 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:30.188+0000 7f0770d55540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T16:04:30.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T16:04:30.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T16:04:30.775 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T16:04:30.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd0 key to auth repository 2024-09-18T16:04:30.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:04:30.950 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: key for osd.0 not found in provided keyring 2024-09-18T16:04:30.952 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T16:04:30.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.0 2024-09-18T16:04:30.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T16:04:30.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:04:30.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T16:04:30.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T16:04:30.982 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:30.972+0000 7f980426d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:30.984 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:30.976+0000 7f980426d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:30.987 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:30.980+0000 7f980426d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:30.989 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:30.980+0000 7f980426d540 -1 Falling back to public interface 2024-09-18T16:04:31.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 0 2024-09-18T16:04:31.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:04:31.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:04:31.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:04:31.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:04:31.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:31.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:04:31.276 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:04:31.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:31.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:31.496 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:31.488+0000 7f980426d540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T16:04:31.496 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:31.488+0000 7f980426d540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T16:04:31.497 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:31.488+0000 7f980426d540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T16:04:31.506 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:31.496+0000 7f980426d540 -1 osd.0 0 log_to_monitors true 2024-09-18T16:04:31.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:32.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:32.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:32.552 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:04:32.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:04:32.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:32.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:32.730 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/479128,v1:127.0.0.1:6801/479128] [v2:127.0.0.1:6802/479128,v1:127.0.0.1:6803/479128] exists,up 11bf0753-351b-4774-8096-59671f506642 2024-09-18T16:04:32.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:04:32.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:04:32.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:04:32.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-18T16:04:32.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:869: unfound_erasure_coded: '[' false = true ']' 2024-09-18T16:04:32.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:872: unfound_erasure_coded: run_osd_filestore td/osd-scrub-repair 1 2024-09-18T16:04:32.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T16:04:32.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T16:04:32.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=1 2024-09-18T16:04:32.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T16:04:32.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/1 2024-09-18T16:04:32.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T16:04:32.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:04:32.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:04:32.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:04:32.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T16:04:32.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T16:04:32.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T16:04:32.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T16:04:32.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T16:04:32.734 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T16:04:32.734 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:04:32.734 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:04:32.734 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:04:32.734 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:04:32.734 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:04:32.736 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:04:32.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:04:32.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T16:04:32.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T16:04:32.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T16:04:32.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T16:04:32.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T16:04:32.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:04:32.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:04:32.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:04:32.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T16:04:32.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T16:04:32.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/1 2024-09-18T16:04:32.737 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T16:04:32.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=84742619-1a8e-4358-81ac-267187c6cff5 2024-09-18T16:04:32.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd1 84742619-1a8e-4358-81ac-267187c6cff5' 2024-09-18T16:04:32.738 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 84742619-1a8e-4358-81ac-267187c6cff5 2024-09-18T16:04:32.738 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T16:04:32.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQAQ+upm1g8OLRAADpoDXq11jV5BZqAl2A3qvw== 2024-09-18T16:04:32.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQAQ+upm1g8OLRAADpoDXq11jV5BZqAl2A3qvw=="}' 2024-09-18T16:04:32.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 84742619-1a8e-4358-81ac-267187c6cff5 -i td/osd-scrub-repair/1/new.json 2024-09-18T16:04:33.027 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:04:33.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/1/new.json 2024-09-18T16:04:33.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAQ+upm1g8OLRAADpoDXq11jV5BZqAl2A3qvw== --osd-uuid 84742619-1a8e-4358-81ac-267187c6cff5 --osd-objectstore=filestore 2024-09-18T16:04:33.078 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:33.068+0000 7fe018430540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:33.094 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:33.084+0000 7fe018430540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:33.096 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:33.088+0000 7fe018430540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:33.580 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:33.572+0000 7fe018430540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T16:04:34.001 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:33.992+0000 7fe018430540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T16:04:34.002 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:33.992+0000 7fe018430540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T16:04:34.002 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:33.992+0000 7fe018430540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T16:04:34.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-18T16:04:34.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T16:04:34.878 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T16:04:34.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd1 key to auth repository 2024-09-18T16:04:34.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:04:35.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.1 2024-09-18T16:04:35.285 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T16:04:35.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:04:35.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T16:04:35.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T16:04:35.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T16:04:35.316 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:35.308+0000 7f5e162bf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:35.318 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:35.308+0000 7f5e162bf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:35.320 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:35.312+0000 7f5e162bf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:35.321 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:35.312+0000 7f5e162bf540 -1 Falling back to public interface 2024-09-18T16:04:35.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 1 2024-09-18T16:04:35.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:04:35.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T16:04:35.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:04:35.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:04:35.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:35.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:04:35.605 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:04:35.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:35.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:04:35.849 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:35.840+0000 7f5e162bf540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T16:04:35.849 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:35.840+0000 7f5e162bf540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T16:04:35.849 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:35.840+0000 7f5e162bf540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T16:04:35.859 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:35.852+0000 7f5e162bf540 -1 osd.1 0 log_to_monitors true 2024-09-18T16:04:35.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:36.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:36.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:36.908 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:04:36.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:04:36.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:36.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:04:37.220 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/479511,v1:127.0.0.1:6811/479511] [v2:127.0.0.1:6812/479511,v1:127.0.0.1:6813/479511] exists,up 84742619-1a8e-4358-81ac-267187c6cff5 2024-09-18T16:04:37.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:04:37.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:04:37.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:04:37.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-18T16:04:37.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:869: unfound_erasure_coded: '[' false = true ']' 2024-09-18T16:04:37.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:872: unfound_erasure_coded: run_osd_filestore td/osd-scrub-repair 2 2024-09-18T16:04:37.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T16:04:37.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T16:04:37.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=2 2024-09-18T16:04:37.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T16:04:37.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/2 2024-09-18T16:04:37.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T16:04:37.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:04:37.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:04:37.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:04:37.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T16:04:37.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T16:04:37.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T16:04:37.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T16:04:37.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T16:04:37.225 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T16:04:37.225 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:04:37.225 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:04:37.225 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:04:37.225 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:04:37.225 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:04:37.225 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:04:37.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:04:37.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T16:04:37.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T16:04:37.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T16:04:37.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T16:04:37.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T16:04:37.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:04:37.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:04:37.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:04:37.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T16:04:37.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T16:04:37.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/2 2024-09-18T16:04:37.227 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T16:04:37.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=dd68baee-9c43-45b8-a88d-9c8c6811a6ef 2024-09-18T16:04:37.228 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 dd68baee-9c43-45b8-a88d-9c8c6811a6ef 2024-09-18T16:04:37.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd1 dd68baee-9c43-45b8-a88d-9c8c6811a6ef' 2024-09-18T16:04:37.229 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T16:04:37.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQAV+upmW4vADhAAhraSot2fClHz1bWpPnE9Mg== 2024-09-18T16:04:37.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQAV+upmW4vADhAAhraSot2fClHz1bWpPnE9Mg=="}' 2024-09-18T16:04:37.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new dd68baee-9c43-45b8-a88d-9c8c6811a6ef -i td/osd-scrub-repair/2/new.json 2024-09-18T16:04:37.652 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:04:37.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/2/new.json 2024-09-18T16:04:37.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAV+upmW4vADhAAhraSot2fClHz1bWpPnE9Mg== --osd-uuid dd68baee-9c43-45b8-a88d-9c8c6811a6ef --osd-objectstore=filestore 2024-09-18T16:04:37.707 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:37.700+0000 7f894710d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:37.710 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:37.700+0000 7f894710d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:37.711 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:37.704+0000 7f894710d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:38.197 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:38.188+0000 7f894710d540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T16:04:38.761 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:38.752+0000 7f894710d540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T16:04:38.761 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:38.752+0000 7f894710d540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T16:04:38.761 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:38.752+0000 7f894710d540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T16:04:39.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-18T16:04:39.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T16:04:39.514 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T16:04:39.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd2 key to auth repository 2024-09-18T16:04:39.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:04:39.895 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: key for osd.2 not found in provided keyring 2024-09-18T16:04:39.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.2 2024-09-18T16:04:39.898 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T16:04:39.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T16:04:39.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:04:39.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T16:04:39.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T16:04:39.927 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:39.920+0000 7fe3d7bae540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:39.930 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:39.920+0000 7fe3d7bae540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:39.931 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:39.924+0000 7fe3d7bae540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:39.932 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:39.924+0000 7fe3d7bae540 -1 Falling back to public interface 2024-09-18T16:04:40.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 2 2024-09-18T16:04:40.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:04:40.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T16:04:40.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:04:40.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:04:40.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:40.288 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:04:40.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:04:40.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:40.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:04:40.520 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:40.512+0000 7fe3d7bae540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T16:04:40.531 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:40.524+0000 7fe3d7bae540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T16:04:40.531 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:40.524+0000 7fe3d7bae540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T16:04:40.552 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:40.544+0000 7fe3d7bae540 -1 osd.2 0 log_to_monitors true 2024-09-18T16:04:40.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:41.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:41.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:41.597 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:04:41.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:04:41.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:41.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:04:41.985 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/479794,v1:127.0.0.1:6819/479794] [v2:127.0.0.1:6820/479794,v1:127.0.0.1:6821/479794] exists,up dd68baee-9c43-45b8-a88d-9c8c6811a6ef 2024-09-18T16:04:41.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:04:41.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:04:41.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:04:41.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-18T16:04:41.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:869: unfound_erasure_coded: '[' false = true ']' 2024-09-18T16:04:41.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:872: unfound_erasure_coded: run_osd_filestore td/osd-scrub-repair 3 2024-09-18T16:04:41.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-scrub-repair 2024-09-18T16:04:41.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T16:04:41.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=3 2024-09-18T16:04:41.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T16:04:41.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-scrub-repair/3 2024-09-18T16:04:41.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T16:04:41.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:04:41.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:04:41.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:04:41.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-18T16:04:41.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-18T16:04:41.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T16:04:41.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T16:04:41.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T16:04:41.989 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T16:04:41.989 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:04:41.989 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:04:41.989 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:04:41.991 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:04:41.991 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:04:41.991 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:04:41.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:04:41.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T16:04:41.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T16:04:41.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T16:04:41.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T16:04:41.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T16:04:41.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:04:41.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:04:41.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:04:41.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T16:04:41.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T16:04:41.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-scrub-repair/3 2024-09-18T16:04:41.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T16:04:41.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=ec621db8-3ffa-4bf7-8a0d-f60d049d177d 2024-09-18T16:04:41.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd1 ec621db8-3ffa-4bf7-8a0d-f60d049d177d' 2024-09-18T16:04:41.995 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 ec621db8-3ffa-4bf7-8a0d-f60d049d177d 2024-09-18T16:04:41.995 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T16:04:42.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQAa+upmuI7HABAA+JHPW7CScm/1nektSp6kQQ== 2024-09-18T16:04:42.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQAa+upmuI7HABAA+JHPW7CScm/1nektSp6kQQ=="}' 2024-09-18T16:04:42.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new ec621db8-3ffa-4bf7-8a0d-f60d049d177d -i td/osd-scrub-repair/3/new.json 2024-09-18T16:04:42.441 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:04:42.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-scrub-repair/3/new.json 2024-09-18T16:04:42.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAa+upmuI7HABAA+JHPW7CScm/1nektSp6kQQ== --osd-uuid ec621db8-3ffa-4bf7-8a0d-f60d049d177d --osd-objectstore=filestore 2024-09-18T16:04:42.491 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:42.484+0000 7fd19d242540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:42.498 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:42.488+0000 7fd19d242540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:42.501 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:42.492+0000 7fd19d242540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:43.144 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:43.136+0000 7fd19d242540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T16:04:43.620 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:43.612+0000 7fd19d242540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T16:04:43.631 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:43.620+0000 7fd19d242540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T16:04:43.631 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:43.620+0000 7fd19d242540 -1 journal do_read_entry(4096): bad header magic 2024-09-18T16:04:44.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-18T16:04:44.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T16:04:44.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd3 key to auth repository 2024-09-18T16:04:44.293 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd3 key to auth repository 2024-09-18T16:04:44.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:04:44.664 INFO:tasks.workunit.client.0.smithi046.stderr:Error EINVAL: key for osd.3 not found in provided keyring 2024-09-18T16:04:44.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.3 2024-09-18T16:04:44.668 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.3 2024-09-18T16:04:44.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:04:44.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T16:04:44.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T16:04:44.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T16:04:44.694 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:44.684+0000 7f74fc36c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:44.696 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:44.688+0000 7f74fc36c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:44.700 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:44.688+0000 7f74fc36c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:44.701 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:44.692+0000 7f74fc36c540 -1 Falling back to public interface 2024-09-18T16:04:45.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 3 2024-09-18T16:04:45.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:04:45.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T16:04:45.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:04:45.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:04:45.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:45.046 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:04:45.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:04:45.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:45.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:04:45.265 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:45.256+0000 7f74fc36c540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T16:04:45.271 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:45.264+0000 7f74fc36c540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T16:04:45.272 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:45.264+0000 7f74fc36c540 -1 journal do_read_entry(8192): bad header magic 2024-09-18T16:04:45.279 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:04:45.272+0000 7f74fc36c540 -1 osd.3 0 log_to_monitors true 2024-09-18T16:04:45.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:46.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:46.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:46.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:04:46.458 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:04:46.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:46.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:04:46.808 INFO:tasks.workunit.client.0.smithi046.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/480077,v1:127.0.0.1:6827/480077] [v2:127.0.0.1:6828/480077,v1:127.0.0.1:6829/480077] exists,up ec621db8-3ffa-4bf7-8a0d-f60d049d177d 2024-09-18T16:04:46.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:04:46.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:04:46.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:04:46.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:876: unfound_erasure_coded: create_ec_pool ecpool false k=2 m=2 2024-09-18T16:04:46.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: create_ec_pool: local pool_name=ecpool 2024-09-18T16:04:46.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: create_ec_pool: shift 2024-09-18T16:04:46.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: create_ec_pool: local allow_overwrites=false 2024-09-18T16:04:46.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: create_ec_pool: shift 2024-09-18T16:04:46.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-09-18T16:04:47.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-18T16:04:47.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-18T16:04:48.455 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'ecpool' created 2024-09-18T16:04:48.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T16:04:49.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: create_ec_pool: '[' false = true ']' 2024-09-18T16:04:49.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: create_ec_pool: wait_for_clean 2024-09-18T16:04:49.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:04:49.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:04:49.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:04:49.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:04:49.472 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:04:49.473 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:04:49.474 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:04:49.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:04:49.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:04:49.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:04:49.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:04:49.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:04:49.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:04:49.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:04:49.723 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:04:50.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:04:50.070 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:04:50.070 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T16:04:50.071 INFO:tasks.workunit.client.0.smithi046.stderr:3' 2024-09-18T16:04:50.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:04:50.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:04:50.071 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:04:50.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836485 2024-09-18T16:04:50.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836485 2024-09-18T16:04:50.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836485' 2024-09-18T16:04:50.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:04:50.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:04:50.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672964 2024-09-18T16:04:50.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672964 2024-09-18T16:04:50.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836485 1-42949672964' 2024-09-18T16:04:50.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:04:50.568 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:04:50.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509443 2024-09-18T16:04:50.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509443 2024-09-18T16:04:50.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836485 1-42949672964 2-64424509443' 2024-09-18T16:04:50.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:04:50.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T16:04:50.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345922 2024-09-18T16:04:50.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345922 2024-09-18T16:04:50.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836485 1-42949672964 2-64424509443 3-85899345922' 2024-09-18T16:04:50.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:04:50.881 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836485 2024-09-18T16:04:50.882 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:04:50.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:04:50.886 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836485 2024-09-18T16:04:50.886 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:04:50.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836485 2024-09-18T16:04:50.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-09-18T16:04:50.888 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836485 2024-09-18T16:04:50.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:04:51.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836485 -lt 21474836485 2024-09-18T16:04:51.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:04:51.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672964 2024-09-18T16:04:51.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:04:51.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:04:51.266 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672964 2024-09-18T16:04:51.266 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:04:51.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672964 2024-09-18T16:04:51.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-09-18T16:04:51.269 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672964 2024-09-18T16:04:51.270 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:04:51.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672963 -lt 42949672964 2024-09-18T16:04:51.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:04:52.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:04:52.618 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:04:52.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672964 -lt 42949672964 2024-09-18T16:04:52.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:04:52.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509443 2024-09-18T16:04:52.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:04:52.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:04:52.936 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509443 2024-09-18T16:04:52.936 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:04:52.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509443 2024-09-18T16:04:52.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2024-09-18T16:04:52.939 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509443 2024-09-18T16:04:52.939 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:04:53.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509444 -lt 64424509443 2024-09-18T16:04:53.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:04:53.276 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345922 2024-09-18T16:04:53.277 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:04:53.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T16:04:53.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345922 2024-09-18T16:04:53.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:04:53.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345922 2024-09-18T16:04:53.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345922' 2024-09-18T16:04:53.283 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.3 seq 85899345922 2024-09-18T16:04:53.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T16:04:53.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345923 -lt 85899345922 2024-09-18T16:04:53.626 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:04:53.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:04:53.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:04:54.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:04:54.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:04:54.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:04:54.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:04:54.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:04:54.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:04:54.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:04:54.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:04:54.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:04:54.396 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:04:54.397 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:04:54.397 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:04:54.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:04:54.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:04:54.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:04:54.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: create_ec_pool: return 0 2024-09-18T16:04:54.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:878: unfound_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-18T16:04:54.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T16:04:54.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2024-09-18T16:04:54.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=SOMETHING 2024-09-18T16:04:54.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T16:04:54.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T16:04:54.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T16:04:55.234 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T16:04:55.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T16:04:55.922 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T16:04:55.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T16:04:55.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T16:04:55.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-18T16:04:56.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:880: unfound_erasure_coded: get_primary ecpool SOMETHING 2024-09-18T16:04:56.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-09-18T16:04:56.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T16:04:56.264 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-18T16:04:56.264 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T16:04:56.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:880: unfound_erasure_coded: local primary=1 2024-09-18T16:04:56.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:881: unfound_erasure_coded: osds=($(get_osds $poolname SOMETHING | sed -e "s/$primary//")) 2024-09-18T16:04:56.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:881: unfound_erasure_coded: get_osds ecpool SOMETHING 2024-09-18T16:04:56.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:881: unfound_erasure_coded: sed -e s/1// 2024-09-18T16:04:56.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-09-18T16:04:56.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-18T16:04:56.625 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-18T16:04:56.625 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T16:04:57.042 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-09-18T16:04:57.042 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T16:04:57.042 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T16:04:57.042 INFO:tasks.workunit.client.0.smithi046.stderr:3' 2024-09-18T16:04:57.042 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 3 2024-09-18T16:04:57.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:881: unfound_erasure_coded: local -a osds 2024-09-18T16:04:57.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:882: unfound_erasure_coded: local not_primary_first=0 2024-09-18T16:04:57.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:883: unfound_erasure_coded: local not_primary_second=2 2024-09-18T16:04:57.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:884: unfound_erasure_coded: local not_primary_third=3 2024-09-18T16:04:57.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:889: unfound_erasure_coded: pids= 2024-09-18T16:04:57.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-18T16:04:57.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T16:04:57.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T16:04:57.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 480831"' 2024-09-18T16:04:57.045 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 480831' 2024-09-18T16:04:57.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-18T16:04:57.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T16:04:57.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T16:04:57.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 480832"' 2024-09-18T16:04:57.046 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 480832' 2024-09-18T16:04:57.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:892: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T16:04:57.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T16:04:57.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-18T16:04:57.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T16:04:57.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/480834: /' 2024-09-18T16:04:57.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/480836: /' 2024-09-18T16:04:57.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-18T16:04:57.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 480837"' 2024-09-18T16:04:57.047 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 480837' 2024-09-18T16:04:57.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T16:04:57.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: unfound_erasure_coded: wait_background pids 2024-09-18T16:04:57.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: wait_background: pids=' 480831 480832 480837' 2024-09-18T16:04:57.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: wait_background: return_code=0 2024-09-18T16:04:57.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T16:04:57.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 480831 2024-09-18T16:04:57.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/480839: /' 2024-09-18T16:04:58.760 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T16:04:58.760 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T16:04:58.760 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-18T16:04:58.760 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T16:04:58.760 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-09-18T16:04:58.760 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T16:04:58.760 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T16:04:58.761 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-18T16:04:58.761 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T16:04:58.761 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-18T16:04:58.761 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:04:58.761 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:04:58.761 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:04:58.761 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:04:58.761 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:04:58.761 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:04:58.761 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-09-18T16:04:58.761 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T16:04:58.761 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T16:04:58.762 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-18T16:04:58.762 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T16:04:58.762 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-18T16:04:58.762 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-09-18T16:04:58.762 INFO:tasks.workunit.client.0.smithi046.stderr:480836: remove 2#1:eb822e21:::SOMETHING:head# 2024-09-18T16:04:58.762 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-18T16:04:58.762 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T16:04:58.762 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T16:04:58.762 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T16:04:58.762 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T16:04:58.762 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-18T16:04:58.762 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T16:04:58.763 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:04:58.763 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: 480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T16:04:58.763 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T16:04:58.763 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-18T16:04:58.763 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T16:04:58.763 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T16:04:58.763 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T16:04:58.763 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T16:04:58.763 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-18T16:04:58.763 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T16:04:58.763 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-18T16:04:58.763 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:04:58.763 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:04:58.764 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:04:58.764 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:04:58.764 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:04:58.764 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:04:58.764 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T16:04:58.764 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T16:04:58.764 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T16:04:58.764 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-18T16:04:58.764 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T16:04:58.764 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-18T16:04:58.764 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-18T16:04:58.764 INFO:tasks.workunit.client.0.smithi046.stderr:480839: remove 3#1:eb822e21:::SOMETHING:head# 2024-09-18T16:04:58.764 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-18T16:04:58.765 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T16:04:58.765 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T16:04:58.765 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T16:04:58.765 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T16:04:58.765 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-18T16:04:58.765 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T16:04:58.765 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:04:58.765 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: 480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T16:04:58.765 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T16:04:58.765 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T16:04:58.765 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T16:04:58.765 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-18T16:04:58.765 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T16:04:58.765 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T16:04:58.766 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T16:04:58.766 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T16:04:58.766 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T16:04:58.766 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:04:58.766 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:04:58.766 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:04:58.766 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:04:58.766 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:04:58.766 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:04:58.766 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-18T16:04:58.766 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T16:04:58.766 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T16:04:58.766 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T16:04:58.767 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T16:04:58.767 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T16:04:58.767 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-18T16:04:58.767 INFO:tasks.workunit.client.0.smithi046.stderr:480834: remove 1#1:eb822e21:::SOMETHING:head# 2024-09-18T16:04:58.767 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-18T16:04:58.767 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T16:04:58.767 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T16:04:58.767 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T16:04:58.767 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T16:04:58.767 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T16:04:58.767 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T16:04:58.767 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:04:58.768 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:04:58.768 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:04:58.768 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T16:04:58.768 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T16:04:58.768 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T16:04:58.768 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T16:04:58.769 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T16:04:58.769 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T16:04:58.769 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:04:58.769 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:04:58.769 INFO:tasks.workunit.client.0.smithi046.stderr:480834: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:04:58.769 INFO:tasks.workunit.client.0.smithi046.stderr:480834: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:04:58.769 INFO:tasks.workunit.client.0.smithi046.stderr:480834: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:04:58.769 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:04:58.769 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:04:58.769 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:04:58.769 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T16:04:58.769 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T16:04:58.769 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:04:58.769 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:04:58.770 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:04:58.770 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:04:58.770 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T16:04:58.770 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T16:04:58.770 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T16:04:58.770 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T16:04:58.770 INFO:tasks.workunit.client.0.smithi046.stderr:480834: start osd.0 2024-09-18T16:04:58.770 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:04:58.770 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:04:58.770 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T16:04:58.770 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T16:04:58.770 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T16:04:58.770 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T16:04:58.771 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T16:04:58.771 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T16:04:58.771 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:04:58.771 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:04:58.771 INFO:tasks.workunit.client.0.smithi046.stderr:480836: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:04:58.771 INFO:tasks.workunit.client.0.smithi046.stderr:480836: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:04:58.771 INFO:tasks.workunit.client.0.smithi046.stderr:480836: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:04:58.771 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:04:58.771 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:04:58.771 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:04:58.771 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T16:04:58.771 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T16:04:58.771 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:04:58.771 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:04:58.772 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:04:58.772 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:04:58.772 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T16:04:58.772 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T16:04:58.772 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-18T16:04:58.772 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T16:04:58.772 INFO:tasks.workunit.client.0.smithi046.stderr:480836: start osd.2 2024-09-18T16:04:58.772 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:04:58.772 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:04:58.772 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-18T16:04:58.772 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-18T16:04:58.772 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T16:04:58.772 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T16:04:58.773 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T16:04:58.773 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T16:04:58.773 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:04:58.773 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:04:58.773 INFO:tasks.workunit.client.0.smithi046.stderr:480839: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:04:58.773 INFO:tasks.workunit.client.0.smithi046.stderr:480839: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:04:58.773 INFO:tasks.workunit.client.0.smithi046.stderr:480839: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:04:58.773 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:04:58.773 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:04:58.773 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:04:58.773 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T16:04:58.773 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T16:04:58.773 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:04:58.773 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:04:58.774 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:04:58.774 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:04:58.774 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T16:04:58.774 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T16:04:58.774 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-18T16:04:58.774 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T16:04:58.774 INFO:tasks.workunit.client.0.smithi046.stderr:480839: start osd.3 2024-09-18T16:05:00.844 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:05:00.844 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T16:05:00.845 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T16:05:00.845 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T16:05:00.845 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T16:05:00.845 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T16:05:00.845 INFO:tasks.workunit.client.0.smithi046.stderr:480834: 2024-09-18T16:04:58.804+0000 7fb2a1296540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:00.845 INFO:tasks.workunit.client.0.smithi046.stderr:480834: 2024-09-18T16:04:58.820+0000 7fb2a1296540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:00.845 INFO:tasks.workunit.client.0.smithi046.stderr:480834: 2024-09-18T16:04:58.820+0000 7fb2a1296540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:00.845 INFO:tasks.workunit.client.0.smithi046.stderr:480834: 2024-09-18T16:04:58.828+0000 7fb2a1296540 -1 Falling back to public interface 2024-09-18T16:05:00.845 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T16:05:00.845 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:05:00.845 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:05:00.845 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:05:00.845 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:05:00.846 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:00.846 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:05:00.846 INFO:tasks.workunit.client.0.smithi046.stderr:480834: 0 2024-09-18T16:05:00.846 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:00.846 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:00.846 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:00.846 INFO:tasks.workunit.client.0.smithi046.stderr:480834: 2024-09-18T16:04:59.528+0000 7fb2a1296540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T16:05:00.846 INFO:tasks.workunit.client.0.smithi046.stderr:480834: 2024-09-18T16:04:59.536+0000 7fb2a1296540 -1 journal do_read_entry(245760): bad header magic 2024-09-18T16:05:00.846 INFO:tasks.workunit.client.0.smithi046.stderr:480834: 2024-09-18T16:04:59.536+0000 7fb2a1296540 -1 journal do_read_entry(245760): bad header magic 2024-09-18T16:05:00.846 INFO:tasks.workunit.client.0.smithi046.stderr:480834: 2024-09-18T16:04:59.556+0000 7fb2a1296540 -1 osd.0 28 log_to_monitors true 2024-09-18T16:05:00.846 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:00.846 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:00.846 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:05:00.847 INFO:tasks.workunit.client.0.smithi046.stderr:480834: 1 2024-09-18T16:05:00.847 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:00.847 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:00.847 INFO:tasks.workunit.client.0.smithi046.stderr:480834: osd.0 up in weight 1 up_from 33 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6826/481060,v1:127.0.0.1:6827/481060] [v2:127.0.0.1:6828/481060,v1:127.0.0.1:6829/481060] exists,up 11bf0753-351b-4774-8096-59671f506642 2024-09-18T16:05:00.847 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:05:00.847 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:05:00.847 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:05:00.847 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T16:05:00.847 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:05:00.847 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:05:00.869 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_cleanops 2024-09-18T16:05:00.869 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-18T16:05:00.869 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T16:05:00.869 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T16:05:00.869 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T16:05:00.869 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T16:05:00.869 INFO:tasks.workunit.client.0.smithi046.stderr:480839: 2024-09-18T16:04:58.800+0000 7fe9bd6c5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:00.870 INFO:tasks.workunit.client.0.smithi046.stderr:480839: 2024-09-18T16:04:58.816+0000 7fe9bd6c5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:00.870 INFO:tasks.workunit.client.0.smithi046.stderr:480839: 2024-09-18T16:04:58.820+0000 7fe9bd6c5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:00.870 INFO:tasks.workunit.client.0.smithi046.stderr:480839: 2024-09-18T16:04:58.824+0000 7fe9bd6c5540 -1 Falling back to public interface 2024-09-18T16:05:00.870 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T16:05:00.870 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:05:00.870 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T16:05:00.870 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:05:00.870 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:05:00.870 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:00.870 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:05:00.870 INFO:tasks.workunit.client.0.smithi046.stderr:480839: 0 2024-09-18T16:05:00.870 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:00.870 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:05:00.870 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:00.871 INFO:tasks.workunit.client.0.smithi046.stderr:480839: 2024-09-18T16:04:59.528+0000 7fe9bd6c5540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T16:05:00.871 INFO:tasks.workunit.client.0.smithi046.stderr:480839: 2024-09-18T16:04:59.536+0000 7fe9bd6c5540 -1 journal do_read_entry(163840): bad header magic 2024-09-18T16:05:00.871 INFO:tasks.workunit.client.0.smithi046.stderr:480839: 2024-09-18T16:04:59.536+0000 7fe9bd6c5540 -1 journal do_read_entry(163840): bad header magic 2024-09-18T16:05:00.871 INFO:tasks.workunit.client.0.smithi046.stderr:480839: 2024-09-18T16:04:59.556+0000 7fe9bd6c5540 -1 osd.3 28 log_to_monitors true 2024-09-18T16:05:00.871 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:00.871 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:00.871 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:05:00.871 INFO:tasks.workunit.client.0.smithi046.stderr:480839: 1 2024-09-18T16:05:00.871 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:00.871 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:05:00.871 INFO:tasks.workunit.client.0.smithi046.stderr:480839: osd.3 up in weight 1 up_from 32 up_thru 0 down_at 29 last_clean_interval [20,28) [v2:127.0.0.1:6805/481053,v1:127.0.0.1:6807/481053] [v2:127.0.0.1:6819/481053,v1:127.0.0.1:6821/481053] exists,up ec621db8-3ffa-4bf7-8a0d-f60d049d177d 2024-09-18T16:05:00.871 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:05:00.871 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:05:00.872 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:05:00.872 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T16:05:00.872 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:05:00.872 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:05:01.003 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_cleaops 2024-09-18T16:05:01.003 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-18T16:05:01.003 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T16:05:01.003 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T16:05:01.004 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T16:05:01.004 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T16:05:01.004 INFO:tasks.workunit.client.0.smithi046.stderr:480836: 2024-09-18T16:04:58.800+0000 7f8380372540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:01.004 INFO:tasks.workunit.client.0.smithi046.stderr:480836: 2024-09-18T16:04:58.816+0000 7f8380372540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:01.004 INFO:tasks.workunit.client.0.smithi046.stderr:480836: 2024-09-18T16:04:58.820+0000 7f8380372540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:01.004 INFO:tasks.workunit.client.0.smithi046.stderr:480836: 2024-09-18T16:04:58.824+0000 7f8380372540 -1 Falling back to public interface 2024-09-18T16:05:01.004 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T16:05:01.004 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:05:01.004 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T16:05:01.004 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:05:01.004 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:05:01.004 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:01.005 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:05:01.005 INFO:tasks.workunit.client.0.smithi046.stderr:480836: 0 2024-09-18T16:05:01.005 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:01.005 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:05:01.005 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:01.005 INFO:tasks.workunit.client.0.smithi046.stderr:480836: 2024-09-18T16:04:59.596+0000 7f8380372540 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T16:05:01.005 INFO:tasks.workunit.client.0.smithi046.stderr:480836: 2024-09-18T16:04:59.604+0000 7f8380372540 -1 journal do_read_entry(192512): bad header magic 2024-09-18T16:05:01.005 INFO:tasks.workunit.client.0.smithi046.stderr:480836: 2024-09-18T16:04:59.604+0000 7f8380372540 -1 journal do_read_entry(192512): bad header magic 2024-09-18T16:05:01.005 INFO:tasks.workunit.client.0.smithi046.stderr:480836: 2024-09-18T16:04:59.612+0000 7f8380372540 -1 osd.2 28 log_to_monitors true 2024-09-18T16:05:01.005 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:01.005 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:01.005 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:05:01.005 INFO:tasks.workunit.client.0.smithi046.stderr:480836: 1 2024-09-18T16:05:01.005 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:01.006 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:05:01.006 INFO:tasks.workunit.client.0.smithi046.stderr:480836: osd.2 up in weight 1 up_from 33 up_thru 0 down_at 29 last_clean_interval [15,28) [v2:127.0.0.1:6800/481057,v1:127.0.0.1:6801/481057] [v2:127.0.0.1:6802/481057,v1:127.0.0.1:6803/481057] exists,up dd68baee-9c43-45b8-a88d-9c8c6811a6ef 2024-09-18T16:05:01.006 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:05:01.006 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:05:01.006 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:05:01.006 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T16:05:01.006 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:05:01.006 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:05:01.891 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clea: local cur_active_clean 2024-09-18T16:05:01.891 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:05:01.891 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:05:01.891 INFO:tasks.workunit.client.0.smithi046.stderr:480834: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:05:01.891 INFO:tasks.workunit.client.0.smithi046.stderr:480834: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:05:01.892 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:05:01.892 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:05:01.892 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:05:01.892 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:05:01.892 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:05:01.892 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:05:01.892 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:05:01.892 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:05:01.892 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:05:01.892 INFO:tasks.workunit.client.0.smithi046.stderr:480834: 1 2024-09-18T16:05:01.893 INFO:tasks.workunit.client.0.smithi046.stderr:480834: 2 2024-09-18T16:05:01.893 INFO:tasks.workunit.client.0.smithi046.stderr:480834: 3' 2024-09-18T16:05:01.893 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:05:01.893 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:05:01.893 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:05:01.893 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=141733920771 2024-09-18T16:05:01.893 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 141733920771 2024-09-18T16:05:01.893 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-141733920771' 2024-09-18T16:05:01.893 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:05:01.893 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:05:01.893 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672967 2024-09-18T16:05:01.893 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672967 2024-09-18T16:05:01.893 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-141733920771 1-42949672967' 2024-09-18T16:05:01.894 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:05:01.894 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:05:01.894 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=141733920771 2024-09-18T16:05:01.894 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 141733920771 2024-09-18T16:05:01.894 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-141733920771 1-42949672967 2-141733920771' 2024-09-18T16:05:01.894 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:05:01.894 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T16:05:01.894 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953474 2024-09-18T16:05:01.894 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953474 2024-09-18T16:05:01.895 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178n: local cur_active_clean 2024-09-18T16:05:01.895 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:05:01.895 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:05:01.895 INFO:tasks.workunit.client.0.smithi046.stderr:480839: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:05:01.895 INFO:tasks.workunit.client.0.smithi046.stderr:480839: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:05:01.895 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:05:01.895 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:05:01.895 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:05:01.895 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:05:01.895 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:05:01.895 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:05:01.895 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:05:01.895 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:05:01.896 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:05:01.896 INFO:tasks.workunit.client.0.smithi046.stderr:480839: 1 2024-09-18T16:05:01.896 INFO:tasks.workunit.client.0.smithi046.stderr:480839: 2 2024-09-18T16:05:01.896 INFO:tasks.workunit.client.0.smithi046.stderr:480839: 3' 2024-09-18T16:05:01.896 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:05:01.896 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:05:01.896 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:05:01.896 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=141733920770 2024-09-18T16:05:01.897 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 141733920770 2024-09-18T16:05:01.897 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-141733920770' 2024-09-18T16:05:01.897 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:05:01.897 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:05:01.897 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672968 2024-09-18T16:05:01.897 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672968 2024-09-18T16:05:01.897 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-141733920770 1-42949672968' 2024-09-18T16:05:01.897 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:05:01.897 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:05:01.897 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=141733920770 2024-09-18T16:05:01.897 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 141733920770 2024-09-18T16:05:01.897 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-141733920770 1-42949672968 2-141733920770' 2024-09-18T16:05:01.898 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:05:01.898 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T16:05:01.898 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953475 2024-09-18T16:05:01.898 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953475 2024-09-18T16:05:02.038 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:217n: local cur_active_clean 2024-09-18T16:05:02.038 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:05:02.039 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:05:02.039 INFO:tasks.workunit.client.0.smithi046.stderr:480836: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:05:02.039 INFO:tasks.workunit.client.0.smithi046.stderr:480836: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:05:02.039 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:05:02.039 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:05:02.039 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:05:02.039 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:05:02.039 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:05:02.039 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:05:02.039 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:05:02.039 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:05:02.039 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:05:02.039 INFO:tasks.workunit.client.0.smithi046.stderr:480836: 1 2024-09-18T16:05:02.040 INFO:tasks.workunit.client.0.smithi046.stderr:480836: 2 2024-09-18T16:05:02.040 INFO:tasks.workunit.client.0.smithi046.stderr:480836: 3' 2024-09-18T16:05:02.040 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:05:02.040 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:05:02.040 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:05:02.040 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=141733920772 2024-09-18T16:05:02.040 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 141733920772 2024-09-18T16:05:02.040 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-141733920772' 2024-09-18T16:05:02.040 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:05:02.040 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:05:02.040 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672969 2024-09-18T16:05:02.040 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672969 2024-09-18T16:05:02.040 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-141733920772 1-42949672969' 2024-09-18T16:05:02.041 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:05:02.041 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:05:02.041 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=141733920772 2024-09-18T16:05:02.041 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 141733920772 2024-09-18T16:05:02.041 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-141733920772 1-42949672969 2-141733920772' 2024-09-18T16:05:02.041 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:05:02.041 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T16:05:02.041 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953476 2024-09-18T16:05:02.041 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953476 2024-09-18T16:05:03.903 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-141733920770 1-42949672968 2-141733920770 3-137438953475' 2024-09-18T16:05:03.903 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:05:03.903 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-141733920770 2024-09-18T16:05:03.903 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:05:03.903 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:05:03.903 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:05:03.903 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-141733920770 2024-09-18T16:05:03.903 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=141733920770 2024-09-18T16:05:03.904 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 141733920770' 2024-09-18T16:05:03.904 INFO:tasks.workunit.client.0.smithi046.stderr:480839: waiting osd.0 seq 141733920770 2024-09-18T16:05:03.904 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:05:03.904 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 141733920770 2024-09-18T16:05:03.904 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:05:03.904 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:05:03.904 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:05:03.904 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 141733920772 -lt 141733920770 2024-09-18T16:05:03.904 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:05:03.904 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672968 2024-09-18T16:05:03.904 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:05:03.904 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:05:03.904 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672968 2024-09-18T16:05:03.905 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:05:03.905 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672968 2024-09-18T16:05:03.905 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-09-18T16:05:03.905 INFO:tasks.workunit.client.0.smithi046.stderr:480839: waiting osd.1 seq 42949672968 2024-09-18T16:05:03.905 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:05:03.905 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672969 -lt 42949672968 2024-09-18T16:05:03.905 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:05:03.905 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-141733920770 2024-09-18T16:05:03.905 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:05:03.905 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:05:03.905 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-141733920770 2024-09-18T16:05:03.905 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:05:03.905 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=141733920770 2024-09-18T16:05:03.905 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 141733920770' 2024-09-18T16:05:03.906 INFO:tasks.workunit.client.0.smithi046.stderr:480839: waiting osd.2 seq 141733920770 2024-09-18T16:05:03.909 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq: flush_pg_stats: seqs=' 0-141733920771 1-42949672967 2-141733920771 3-137438953474' 2024-09-18T16:05:03.909 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:05:03.909 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:05:03.909 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-141733920771 2024-09-18T16:05:03.909 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:05:03.910 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-141733920771 2024-09-18T16:05:03.910 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:05:03.910 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=141733920771 2024-09-18T16:05:03.910 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 141733920771' 2024-09-18T16:05:03.910 INFO:tasks.workunit.client.0.smithi046.stderr:480834: waiting osd.0 seq 141733920771 2024-09-18T16:05:03.910 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:05:03.910 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 141733920771 2024-09-18T16:05:03.910 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:05:03.910 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:05:03.910 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:05:03.910 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 141733920772 -lt 141733920771 2024-09-18T16:05:03.910 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:05:03.910 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672967 2024-09-18T16:05:03.911 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:05:03.911 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:05:03.911 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672967 2024-09-18T16:05:03.911 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:05:03.911 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672967 2024-09-18T16:05:03.911 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-09-18T16:05:03.911 INFO:tasks.workunit.client.0.smithi046.stderr:480834: waiting osd.1 seq 42949672967 2024-09-18T16:05:03.911 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:05:03.911 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672969 -lt 42949672967 2024-09-18T16:05:03.911 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:05:03.911 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-141733920771 2024-09-18T16:05:03.911 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:05:03.911 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:05:03.912 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-141733920771 2024-09-18T16:05:03.912 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:05:03.912 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=141733920771 2024-09-18T16:05:03.912 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 141733920771' 2024-09-18T16:05:03.912 INFO:tasks.workunit.client.0.smithi046.stderr:480834: waiting osd.2 seq 141733920771 2024-09-18T16:05:04.058 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8: flush_pg_stats: seqs=' 0-141733920772 1-42949672969 2-141733920772 3-137438953476' 2024-09-18T16:05:04.058 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:05:04.058 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-141733920772 2024-09-18T16:05:04.058 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:05:04.059 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:05:04.059 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:05:04.059 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-141733920772 2024-09-18T16:05:04.059 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=141733920772 2024-09-18T16:05:04.059 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 141733920772' 2024-09-18T16:05:04.059 INFO:tasks.workunit.client.0.smithi046.stderr:480836: waiting osd.0 seq 141733920772 2024-09-18T16:05:04.059 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:05:04.059 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 141733920772 2024-09-18T16:05:04.059 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:05:04.059 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:05:04.059 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:05:04.059 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 141733920772 -lt 141733920772 2024-09-18T16:05:04.059 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:05:04.060 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672969 2024-09-18T16:05:04.060 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:05:04.060 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:05:04.060 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672969 2024-09-18T16:05:04.060 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:05:04.060 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672969 2024-09-18T16:05:04.060 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-09-18T16:05:04.060 INFO:tasks.workunit.client.0.smithi046.stderr:480836: waiting osd.1 seq 42949672969 2024-09-18T16:05:04.060 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:05:04.060 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672969 -lt 42949672969 2024-09-18T16:05:04.060 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:05:04.060 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-141733920772 2024-09-18T16:05:04.060 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:05:04.061 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:05:04.061 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-141733920772 2024-09-18T16:05:04.061 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:05:04.061 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=141733920772 2024-09-18T16:05:04.061 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 141733920772' 2024-09-18T16:05:04.061 INFO:tasks.workunit.client.0.smithi046.stderr:480836: waiting osd.2 seq 141733920772 2024-09-18T16:05:05.754 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:05:05.754 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 141733920772 -lt 141733920770 2024-09-18T16:05:05.754 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:05:05.754 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-137438953475 2024-09-18T16:05:05.754 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:05:05.754 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T16:05:05.755 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-137438953475 2024-09-18T16:05:05.755 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:05:05.755 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953475 2024-09-18T16:05:05.755 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 137438953475' 2024-09-18T16:05:05.755 INFO:tasks.workunit.client.0.smithi046.stderr:480839: waiting osd.3 seq 137438953475 2024-09-18T16:05:05.755 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T16:05:05.755 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953476 -lt 137438953475 2024-09-18T16:05:05.755 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:05:05.755 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:05:05.755 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:05:05.755 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:05:05.755 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:05:05.755 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:05:05.755 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:05:05.756 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:05:05.756 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:05:05.756 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:05:05.756 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:05:05.756 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:05:05.756 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:05:05.756 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:05:05.756 INFO:tasks.workunit.client.0.smithi046.stderr:480839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:05:05.756 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:05:05.756 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:05:05.756 INFO:tasks.workunit.client.0.smithi046.stderr:480839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:05:05.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T16:05:05.790 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T16:05:05.791 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 141733920772 -lt 141733920771 2024-09-18T16:05:05.791 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:05:05.791 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-137438953474 2024-09-18T16:05:05.791 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:05:05.791 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T16:05:05.791 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-137438953474 2024-09-18T16:05:05.791 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:05:05.791 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953474 2024-09-18T16:05:05.791 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 137438953474' 2024-09-18T16:05:05.791 INFO:tasks.workunit.client.0.smithi046.stderr:480834: waiting osd.3 seq 137438953474 2024-09-18T16:05:05.791 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T16:05:05.791 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953476 -lt 137438953474 2024-09-18T16:05:05.791 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:05:05.792 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:05:05.792 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:05:05.792 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:05:05.792 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:05:05.792 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:05:05.792 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:05:05.792 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:05:05.792 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:05:05.792 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:05:05.792 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:05:05.792 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:05:05.792 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:05:05.793 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:05:05.793 INFO:tasks.workunit.client.0.smithi046.stderr:480834: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:05:05.793 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:05:05.793 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:05:05.793 INFO:tasks.workunit.client.0.smithi046.stderr:480834: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:05:05.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T16:05:05.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T16:05:05.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 480832 2024-09-18T16:05:06.003 INFO:tasks.workunit.client.0.smithi046.stderr: 2 2024-09-18T16:05:06.003 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 141733920772 -lt 141733920772 2024-09-18T16:05:06.003 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:05:06.003 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-137438953476 2024-09-18T16:05:06.003 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:05:06.003 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T16:05:06.003 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-137438953476 2024-09-18T16:05:06.003 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:05:06.003 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953476 2024-09-18T16:05:06.004 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 137438953476' 2024-09-18T16:05:06.004 INFO:tasks.workunit.client.0.smithi046.stderr:480836: waiting osd.3 seq 137438953476 2024-09-18T16:05:06.004 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T16:05:06.004 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953476 -lt 137438953476 2024-09-18T16:05:06.004 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:05:06.004 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:05:06.004 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:05:06.004 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:05:06.004 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:05:06.004 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:05:06.004 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:05:06.004 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:05:06.004 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:05:06.004 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:05:06.005 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:05:06.005 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:05:06.005 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:05:06.005 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:05:06.005 INFO:tasks.workunit.client.0.smithi046.stderr:480836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:05:06.005 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:05:06.005 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:05:06.005 INFO:tasks.workunit.client.0.smithi046.stderr:480836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:05:06.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T16:05:06.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T16:05:06.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 480837 2024-09-18T16:05:06.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: eval 'pids='\'''\''' 2024-09-18T16:05:06.007 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: pids= 2024-09-18T16:05:06.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: wait_background: return 0 2024-09-18T16:05:06.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: return_code=0 2024-09-18T16:05:06.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:895: unfound_erasure_coded: '[' 0 -ne 0 ']' 2024-09-18T16:05:06.007 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:900: unfound_erasure_coded: get_pg ecpool SOMETHING 2024-09-18T16:05:06.007 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-09-18T16:05:06.007 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T16:05:06.007 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-18T16:05:06.007 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T16:05:06.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:900: unfound_erasure_coded: local pg=1.0 2024-09-18T16:05:06.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:901: unfound_erasure_coded: repair 1.0 2024-09-18T16:05:06.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=1.0 2024-09-18T16:05:06.328 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 1.0 2024-09-18T16:05:06.328 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:05:06.328 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:05:06.329 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:05:06.329 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:05:06.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T16:04:48.267043+0000 2024-09-18T16:05:06.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 1.0 2024-09-18T16:05:06.954 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-09-18T16:05:06.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 1.0 2024-09-18T16:04:48.267043+0000 2024-09-18T16:05:06.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-18T16:05:06.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T16:04:48.267043+0000 2024-09-18T16:05:06.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T16:05:06.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T16:05:06.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:05:06.963 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:05:06.963 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:05:06.963 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:05:06.963 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:05:06.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:05:07.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:04:48.267043+0000 '>' 2024-09-18T16:04:48.267043+0000 2024-09-18T16:05:07.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:05:08.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:05:08.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:05:08.283 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:05:08.283 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:05:08.283 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:05:08.283 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:05:08.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:05:08.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:04:48.267043+0000 '>' 2024-09-18T16:04:48.267043+0000 2024-09-18T16:05:08.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:05:09.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:05:09.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:05:09.604 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:05:09.604 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:05:09.605 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:05:09.605 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:05:09.605 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:05:09.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:04:48.267043+0000 '>' 2024-09-18T16:04:48.267043+0000 2024-09-18T16:05:09.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:05:10.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:05:10.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:05:10.919 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:05:10.919 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:05:10.919 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:05:10.919 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:05:10.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:05:11.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:04:48.267043+0000 '>' 2024-09-18T16:04:48.267043+0000 2024-09-18T16:05:11.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:05:12.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:05:12.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:05:12.283 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:05:12.283 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:05:12.283 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:05:12.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:05:12.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:05:12.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:04:48.267043+0000 '>' 2024-09-18T16:04:48.267043+0000 2024-09-18T16:05:12.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:05:13.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:05:13.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:05:13.599 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:05:13.599 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:05:13.599 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:05:13.599 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:05:13.600 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:05:13.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:04:48.267043+0000 '>' 2024-09-18T16:04:48.267043+0000 2024-09-18T16:05:13.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:05:14.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:05:14.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:05:14.916 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:05:14.916 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:05:14.916 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:05:14.917 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:05:14.917 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:05:15.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:05:07.929448+0000 '>' 2024-09-18T16:04:48.267043+0000 2024-09-18T16:05:15.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T16:05:15.282 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:906: unfound_erasure_coded: seq 1 60 2024-09-18T16:05:15.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:906: unfound_erasure_coded: for f in `seq 1 60` 2024-09-18T16:05:15.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:907: unfound_erasure_coded: ceph -s 2024-09-18T16:05:15.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:907: unfound_erasure_coded: grep '1/1 objects unfound' 2024-09-18T16:05:15.641 INFO:tasks.workunit.client.0.smithi046.stdout: 1/1 objects unfound (100.000%) 2024-09-18T16:05:15.641 INFO:tasks.workunit.client.0.smithi046.stdout: 1/1 objects unfound (100.000%) 2024-09-18T16:05:15.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:907: unfound_erasure_coded: break 2024-09-18T16:05:15.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:910: unfound_erasure_coded: ceph -s 2024-09-18T16:05:15.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:910: unfound_erasure_coded: grep '4 up' 2024-09-18T16:05:16.088 INFO:tasks.workunit.client.0.smithi046.stdout: osd: 4 osds: 4 up (since 16s), 4 in (since 33s) 2024-09-18T16:05:16.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:911: unfound_erasure_coded: ceph -s 2024-09-18T16:05:16.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:911: unfound_erasure_coded: grep '4 in' 2024-09-18T16:05:16.497 INFO:tasks.workunit.client.0.smithi046.stdout: osd: 4 osds: 4 up (since 16s), 4 in (since 34s) 2024-09-18T16:05:16.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:912: unfound_erasure_coded: ceph -s 2024-09-18T16:05:16.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:912: unfound_erasure_coded: grep '1/1 objects unfound' 2024-09-18T16:05:16.858 INFO:tasks.workunit.client.0.smithi046.stdout: 1/1 objects unfound (100.000%) 2024-09-18T16:05:16.858 INFO:tasks.workunit.client.0.smithi046.stdout: 1/1 objects unfound (100.000%) 2024-09-18T16:05:16.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T16:05:16.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T16:05:16.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:05:16.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T16:05:16.860 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:05:16.860 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:05:16.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:05:16.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:05:16.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:05:16.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:05:16.984 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:05:16.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:05:16.987 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:05:16.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:05:16.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:05:16.991 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:05:16.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:05:16.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:05:16.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:05:16.995 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:05:16.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:05:16.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:05:16.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T16:05:17.015 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:05:17.015 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:05:17.015 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:05:17.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T16:05:17.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:05:17.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:05:17.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2024-09-18T16:05:17.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: setup td/osd-scrub-repair 2024-09-18T16:05:17.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-09-18T16:05:17.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-09-18T16:05:17.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T16:05:17.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:05:17.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T16:05:17.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:05:17.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:05:17.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:05:17.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:05:17.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:05:17.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:05:17.025 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:05:17.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:05:17.028 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:05:17.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:05:17.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:05:17.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:05:17.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:05:17.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:05:17.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:05:17.034 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:05:17.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:05:17.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:05:17.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T16:05:17.040 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:05:17.040 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:05:17.040 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:05:17.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T16:05:17.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:05:17.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:05:17.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-09-18T16:05:17.045 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T16:05:17.045 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:05:17.045 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:05:17.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137209 2024-09-18T16:05:17.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T16:05:17.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T16:05:17.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T16:05:17.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-18T16:05:17.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: TEST_unfound_erasure_coded_overwrites td/osd-scrub-repair 2024-09-18T16:05:17.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: TEST_unfound_erasure_coded_overwrites: '[' true = true ']' 2024-09-18T16:05:17.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: TEST_unfound_erasure_coded_overwrites: unfound_erasure_coded td/osd-scrub-repair true 2024-09-18T16:05:17.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:861: unfound_erasure_coded: local dir=td/osd-scrub-repair 2024-09-18T16:05:17.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:862: unfound_erasure_coded: local allow_overwrites=true 2024-09-18T16:05:17.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: unfound_erasure_coded: local poolname=ecpool 2024-09-18T16:05:17.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: unfound_erasure_coded: local payload=ABCDEF 2024-09-18T16:05:17.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:866: unfound_erasure_coded: run_mon td/osd-scrub-repair a 2024-09-18T16:05:17.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-09-18T16:05:17.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T16:05:17.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T16:05:17.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T16:05:17.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-09-18T16:05:17.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-18T16:05:17.417 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T16:05:17.417 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:05:17.417 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:05:17.417 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:05:17.417 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:05:17.418 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:05:17.418 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:05:17.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T16:05:17.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T16:05:17.716 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T16:05:17.716 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:05:17.716 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:05:17.716 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T16:05:17.717 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T16:05:17.717 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:05:17.718 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:05:17.718 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:05:17.718 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:05:17.719 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:05:17.719 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:05:17.719 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T16:05:17.719 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:05:17.719 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get fsid 2024-09-18T16:05:17.814 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T16:05:17.814 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:05:17.814 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:05:17.814 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T16:05:17.815 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T16:05:17.815 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:05:17.815 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:05:17.815 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:05:17.816 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:05:17.816 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:05:17.816 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:05:17.816 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137209/ceph-mon.a.asok 2024-09-18T16:05:17.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:05:17.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.137209/ceph-mon.a.asok config get mon_host 2024-09-18T16:05:17.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: unfound_erasure_coded: run_mgr td/osd-scrub-repair x 2024-09-18T16:05:17.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-09-18T16:05:17.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T16:05:17.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T16:05:17.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T16:05:17.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-09-18T16:05:17.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T16:05:18.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T16:05:18.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:05:18.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:05:18.758 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:05:18.758 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:05:18.758 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:05:18.758 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:05:18.760 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:05:18.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:05:18.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: seq 0 3 2024-09-18T16:05:18.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-18T16:05:18.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:869: unfound_erasure_coded: '[' true = true ']' 2024-09-18T16:05:18.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:870: unfound_erasure_coded: run_osd td/osd-scrub-repair 0 2024-09-18T16:05:18.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T16:05:18.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:05:18.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T16:05:18.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:05:18.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T16:05:18.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T16:05:18.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:05:18.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:05:18.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:05:18.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T16:05:18.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T16:05:18.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:05:18.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:05:18.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T16:05:18.799 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:05:18.799 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:05:18.799 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:05:18.799 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:05:18.799 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:05:18.799 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:05:18.800 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:05:18.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:05:18.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:05:18.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:05:18.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:05:18.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T16:05:18.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T16:05:18.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:05:18.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:05:18.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:05:18.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:05:18.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:05:18.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:05:18.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T16:05:18.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:05:18.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=773229d1-e50b-4e96-a576-954bbcfa4f77 2024-09-18T16:05:18.814 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 773229d1-e50b-4e96-a576-954bbcfa4f77 2024-09-18T16:05:18.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 773229d1-e50b-4e96-a576-954bbcfa4f77' 2024-09-18T16:05:18.815 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:05:18.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA++upm68yGMRAAm/92BMxx34agiy/STTM34Q== 2024-09-18T16:05:18.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA++upm68yGMRAAm/92BMxx34agiy/STTM34Q=="}' 2024-09-18T16:05:18.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 773229d1-e50b-4e96-a576-954bbcfa4f77 -i td/osd-scrub-repair/0/new.json 2024-09-18T16:05:19.099 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:05:19.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-18T16:05:19.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA++upm68yGMRAAm/92BMxx34agiy/STTM34Q== --osd-uuid 773229d1-e50b-4e96-a576-954bbcfa4f77 2024-09-18T16:05:19.144 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:05:19.136+0000 7f3293858540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:19.192 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:05:19.184+0000 7f3293858540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:19.194 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:05:19.184+0000 7f3293858540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:19.195 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:05:19.188+0000 7f3293858540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T16:05:19.195 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:05:19.188+0000 7f3293858540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T16:05:19.195 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:05:19.188+0000 7f3293858540 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-09-18T16:05:19.239 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:05:19.232+0000 7f3293858540 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-18T16:05:25.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-18T16:05:25.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:05:25.534 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T16:05:25.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T16:05:25.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:05:25.907 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T16:05:25.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T16:05:25.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:05:25.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:05:25.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:05:25.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:05:25.937 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:05:25.928+0000 7fc623172540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:25.940 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:05:25.932+0000 7fc623172540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:25.942 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:05:25.932+0000 7fc623172540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:26.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T16:05:26.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:05:26.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:05:26.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:05:26.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:05:26.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:26.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:05:26.253 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:05:26.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:26.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:26.488 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:05:26.480+0000 7fc623172540 -1 Falling back to public interface 2024-09-18T16:05:26.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:27.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:27.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:27.575 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:05:27.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:05:27.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:27.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:27.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:28.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:28.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:28.888 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:05:28.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:05:28.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:28.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:29.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:29.201 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:05:29.196+0000 7fc623172540 -1 osd.0 0 log_to_monitors true 2024-09-18T16:05:30.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:30.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:30.200 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:05:30.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:05:30.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:30.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:30.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:31.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:31.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:31.512 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:05:31.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:05:31.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:31.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:31.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:32.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:32.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:32.961 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:05:32.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:05:32.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:32.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:33.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:34.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:34.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:34.268 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:05:34.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:05:34.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:34.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:34.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:35.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:35.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:35.579 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:05:35.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:05:35.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:35.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:35.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:36.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:36.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:36.984 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:05:36.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:05:36.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:36.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:37.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:38.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:38.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:38.296 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:05:38.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:05:38.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:38.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:38.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:39.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:39.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:39.612 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:05:39.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:05:39.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:39.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:39.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:40.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:40.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:40.948 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:05:40.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:05:40.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:40.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:41.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:42.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:42.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:42.320 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:05:42.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:05:42.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:42.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:42.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:43.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:43.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:43.916 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:05:43.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:05:43.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:43.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:44.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:45.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:45.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:45.340 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:05:45.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:05:45.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:45.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:46.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:47.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:47.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:47.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:05:47.032 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:05:47.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:47.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:47.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:48.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:48.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:48.460 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:05:48.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:05:48.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:48.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:49.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:50.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:50.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:50.143 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:05:50.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:05:50.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:50.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:50.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:51.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:51.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:51.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:05:51.645 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:05:51.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:51.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:51.957 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/484317,v1:127.0.0.1:6803/484317] [v2:127.0.0.1:6804/484317,v1:127.0.0.1:6805/484317] exists,up 773229d1-e50b-4e96-a576-954bbcfa4f77 2024-09-18T16:05:51.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:05:51.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:05:51.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:05:51.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-18T16:05:51.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:869: unfound_erasure_coded: '[' true = true ']' 2024-09-18T16:05:51.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:870: unfound_erasure_coded: run_osd td/osd-scrub-repair 1 2024-09-18T16:05:51.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T16:05:51.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:05:51.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T16:05:51.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:05:51.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-18T16:05:51.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T16:05:51.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:05:51.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:05:51.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:05:51.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-18T16:05:51.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-18T16:05:51.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:05:51.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:05:51.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T16:05:51.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:05:51.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:05:51.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:05:51.961 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:05:51.962 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:05:51.962 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:05:51.962 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:05:51.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:05:51.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:05:51.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:05:51.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:05:51.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T16:05:51.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T16:05:51.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:05:51.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:05:51.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:05:51.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:05:51.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:05:51.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:05:51.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-18T16:05:51.965 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:05:51.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a35e63e7-ac46-4bb1-8394-d30d530888aa 2024-09-18T16:05:51.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a35e63e7-ac46-4bb1-8394-d30d530888aa' 2024-09-18T16:05:51.968 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 a35e63e7-ac46-4bb1-8394-d30d530888aa 2024-09-18T16:05:51.969 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:05:51.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBf+upmbSXGOhAAkOAEt09dNcl7UZ9iUAHboA== 2024-09-18T16:05:51.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBf+upmbSXGOhAAkOAEt09dNcl7UZ9iUAHboA=="}' 2024-09-18T16:05:51.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a35e63e7-ac46-4bb1-8394-d30d530888aa -i td/osd-scrub-repair/1/new.json 2024-09-18T16:05:52.377 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:05:52.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-18T16:05:52.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBf+upmbSXGOhAAkOAEt09dNcl7UZ9iUAHboA== --osd-uuid a35e63e7-ac46-4bb1-8394-d30d530888aa 2024-09-18T16:05:52.428 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:05:52.423+0000 7f3ab8c67540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:52.434 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:05:52.427+0000 7f3ab8c67540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:52.436 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:05:52.431+0000 7f3ab8c67540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:52.437 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:05:52.431+0000 7f3ab8c67540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T16:05:52.437 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:05:52.431+0000 7f3ab8c67540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T16:05:52.438 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:05:52.431+0000 7f3ab8c67540 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-09-18T16:05:52.507 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:05:52.503+0000 7f3ab8c67540 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-18T16:05:58.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-18T16:05:58.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:05:58.419 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T16:05:58.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T16:05:58.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:05:58.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T16:05:58.809 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T16:05:58.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:05:58.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:05:58.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:05:58.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:05:58.838 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:05:58.831+0000 7efc41e93540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:58.841 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:05:58.835+0000 7efc41e93540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:58.842 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:05:58.835+0000 7efc41e93540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:59.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T16:05:59.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:05:59.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T16:05:59.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:05:59.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:05:59.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:59.150 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:05:59.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:05:59.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:59.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:05:59.404 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:05:59.399+0000 7efc41e93540 -1 Falling back to public interface 2024-09-18T16:05:59.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:00.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:00.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:00.509 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:06:00.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:06:00.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:00.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:00.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:01.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:01.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:01.819 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:06:01.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:06:01.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:01.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:02.086 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:06:02.079+0000 7efc41e93540 -1 osd.1 0 log_to_monitors true 2024-09-18T16:06:02.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:03.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:03.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:03.149 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:06:03.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:06:03.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:03.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:03.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:04.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:04.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:04.460 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:06:04.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:06:04.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:04.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:04.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:05.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:05.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:05.873 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:06:05.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:06:05.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:05.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:06.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:07.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:07.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:07.268 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:06:07.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:06:07.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:07.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:07.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:08.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:08.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:08.588 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:06:08.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:06:08.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:08.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:08.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:09.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:09.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:09.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:06:09.902 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:06:09.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:09.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:10.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:11.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:11.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:11.221 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:06:11.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:06:11.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:11.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:11.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:12.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:12.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:12.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:06:12.540 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:06:12.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:12.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:12.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:13.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:13.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:13.909 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:06:13.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:06:13.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:13.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:14.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:15.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:15.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:15.221 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:06:15.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:06:15.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:15.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:15.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:16.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:16.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:16.676 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:06:16.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:06:16.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:16.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:17.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:18.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:18.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:18.159 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:06:18.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:06:18.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:18.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:18.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:19.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:19.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:19.901 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:06:19.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:06:19.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:19.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:20.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:21.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:21.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:21.349 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:06:21.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:06:21.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:21.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:21.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:22.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:22.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:22.753 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:06:22.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:06:22.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:22.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:23.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:24.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:24.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:24.226 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:06:24.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:06:24.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:24.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:24.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:25.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:25.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:25.729 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T16:06:25.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T16:06:25.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:25.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:26.042 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/486104,v1:127.0.0.1:6811/486104] [v2:127.0.0.1:6812/486104,v1:127.0.0.1:6813/486104] exists,up a35e63e7-ac46-4bb1-8394-d30d530888aa 2024-09-18T16:06:26.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:06:26.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:06:26.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:06:26.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-18T16:06:26.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:869: unfound_erasure_coded: '[' true = true ']' 2024-09-18T16:06:26.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:870: unfound_erasure_coded: run_osd td/osd-scrub-repair 2 2024-09-18T16:06:26.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T16:06:26.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:06:26.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T16:06:26.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:06:26.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-18T16:06:26.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T16:06:26.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:06:26.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:06:26.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:06:26.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T16:06:26.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T16:06:26.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:06:26.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:06:26.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T16:06:26.048 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:06:26.048 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:06:26.048 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:06:26.048 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:06:26.048 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:06:26.048 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:06:26.048 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:06:26.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:06:26.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:06:26.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:06:26.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:06:26.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T16:06:26.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T16:06:26.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:06:26.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:06:26.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:06:26.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:06:26.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:06:26.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:06:26.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-18T16:06:26.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:06:26.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a2f23771-2d72-49c9-b0d1-4d58d31d6f51 2024-09-18T16:06:26.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 a2f23771-2d72-49c9-b0d1-4d58d31d6f51' 2024-09-18T16:06:26.051 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 a2f23771-2d72-49c9-b0d1-4d58d31d6f51 2024-09-18T16:06:26.052 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:06:26.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCC+upmVKAmBBAAyvi1ngR0MM+4kGTHUWfKCQ== 2024-09-18T16:06:26.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCC+upmVKAmBBAAyvi1ngR0MM+4kGTHUWfKCQ=="}' 2024-09-18T16:06:26.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a2f23771-2d72-49c9-b0d1-4d58d31d6f51 -i td/osd-scrub-repair/2/new.json 2024-09-18T16:06:26.501 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:06:26.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-18T16:06:26.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCC+upmVKAmBBAAyvi1ngR0MM+4kGTHUWfKCQ== --osd-uuid a2f23771-2d72-49c9-b0d1-4d58d31d6f51 2024-09-18T16:06:26.551 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:06:26.547+0000 7f34bb18b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:06:26.559 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:06:26.555+0000 7f34bb18b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:06:26.560 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:06:26.555+0000 7f34bb18b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:06:26.561 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:06:26.555+0000 7f34bb18b540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T16:06:26.561 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:06:26.555+0000 7f34bb18b540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T16:06:26.562 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:06:26.555+0000 7f34bb18b540 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-09-18T16:06:26.633 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:06:26.627+0000 7f34bb18b540 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-18T16:06:32.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-18T16:06:32.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:06:32.889 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T16:06:32.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T16:06:32.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:06:33.273 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T16:06:33.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T16:06:33.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:06:33.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:06:33.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:06:33.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:06:33.305 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:06:33.299+0000 7f1ba364e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:06:33.307 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:06:33.303+0000 7f1ba364e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:06:33.309 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:06:33.303+0000 7f1ba364e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:06:33.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T16:06:33.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:06:33.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T16:06:33.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:06:33.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:06:33.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:33.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:06:33.624 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:06:33.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:33.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:06:33.853 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:06:33.847+0000 7f1ba364e540 -1 Falling back to public interface 2024-09-18T16:06:33.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:34.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:34.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:34.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:06:34.976 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:06:34.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:34.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:06:35.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:36.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:36.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:36.281 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:06:36.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:06:36.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:36.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:06:36.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:36.639 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:06:36.635+0000 7f1ba364e540 -1 osd.2 0 log_to_monitors true 2024-09-18T16:06:37.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:37.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:37.604 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:06:37.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:06:37.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:37.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:06:37.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:38.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:38.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:38.924 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:06:38.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:06:38.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:38.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:06:39.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:40.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:40.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:40.323 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:06:40.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:06:40.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:40.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:06:40.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:41.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:41.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:41.645 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:06:41.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:06:41.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:41.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:06:41.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:42.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:42.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:42.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:06:42.965 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:06:42.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:42.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:06:43.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:44.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:44.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:44.304 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:06:44.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:06:44.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:44.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:06:44.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:45.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:45.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:45.616 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:06:45.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:06:45.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:45.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:06:45.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:46.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:46.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:46.939 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:06:46.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:06:46.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:46.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:06:47.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:48.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:48.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:48.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:06:48.326 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:06:48.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:48.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:06:48.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:49.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:49.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:49.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:06:49.643 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:06:49.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:49.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:06:50.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:51.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:51.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:51.129 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:06:51.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:06:51.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:51.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:06:51.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:52.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:52.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:52.858 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:06:52.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:06:52.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:52.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:06:53.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:54.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:54.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:54.365 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:06:54.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:06:54.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:54.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:06:55.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:56.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:56.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:56.113 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:06:56.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:06:56.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:56.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:06:56.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:57.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:57.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:57.622 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:06:57.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:06:57.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:57.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:06:58.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:59.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:59.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:59.058 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:06:59.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:06:59.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:06:59.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:59.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:00.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:00.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:00.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T16:07:00.582 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T16:07:00.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:00.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:07:00.916 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/487908,v1:127.0.0.1:6819/487908] [v2:127.0.0.1:6820/487908,v1:127.0.0.1:6821/487908] exists,up a2f23771-2d72-49c9-b0d1-4d58d31d6f51 2024-09-18T16:07:00.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:07:00.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:07:00.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:07:00.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-18T16:07:00.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:869: unfound_erasure_coded: '[' true = true ']' 2024-09-18T16:07:00.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:870: unfound_erasure_coded: run_osd td/osd-scrub-repair 3 2024-09-18T16:07:00.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-09-18T16:07:00.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:07:00.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T16:07:00.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:07:00.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-09-18T16:07:00.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T16:07:00.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:07:00.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:07:00.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:07:00.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-18T16:07:00.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-18T16:07:00.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:07:00.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:07:00.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T16:07:00.925 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:07:00.925 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:07:00.926 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:07:00.928 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:07:00.928 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:07:00.928 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:07:00.928 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:07:00.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:07:00.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:07:00.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:07:00.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:07:00.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T16:07:00.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T16:07:00.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:07:00.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:07:00.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:07:00.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:07:00.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:07:00.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:07:00.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-09-18T16:07:00.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:07:00.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3b69bade-6a43-4474-9a9c-165b08807eaf 2024-09-18T16:07:00.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 3b69bade-6a43-4474-9a9c-165b08807eaf' 2024-09-18T16:07:00.930 INFO:tasks.workunit.client.0.smithi046.stdout:add osd3 3b69bade-6a43-4474-9a9c-165b08807eaf 2024-09-18T16:07:00.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:07:00.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCk+upmHU9QOBAAl+2uX2eGdPnvODbCbZ5U0Q== 2024-09-18T16:07:00.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCk+upmHU9QOBAAl+2uX2eGdPnvODbCbZ5U0Q=="}' 2024-09-18T16:07:00.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3b69bade-6a43-4474-9a9c-165b08807eaf -i td/osd-scrub-repair/3/new.json 2024-09-18T16:07:01.340 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:07:01.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-09-18T16:07:01.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCk+upmHU9QOBAAl+2uX2eGdPnvODbCbZ5U0Q== --osd-uuid 3b69bade-6a43-4474-9a9c-165b08807eaf 2024-09-18T16:07:01.389 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:07:01.383+0000 7f4623c1d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:07:01.398 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:07:01.391+0000 7f4623c1d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:07:01.399 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:07:01.395+0000 7f4623c1d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:07:01.400 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:07:01.395+0000 7f4623c1d540 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-09-18T16:07:01.400 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:07:01.395+0000 7f4623c1d540 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-09-18T16:07:01.400 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:07:01.395+0000 7f4623c1d540 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-09-18T16:07:01.458 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:07:01.455+0000 7f4623c1d540 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-09-18T16:07:07.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-18T16:07:07.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:07:07.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T16:07:07.539 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd3 key to auth repository 2024-09-18T16:07:07.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:07:07.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T16:07:07.923 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.3 2024-09-18T16:07:07.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:07:07.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:07:07.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:07:07.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:07:07.953 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:07:07.947+0000 7f89d3051540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:07:07.956 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:07:07.951+0000 7f89d3051540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:07:07.958 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:07:07.951+0000 7f89d3051540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:07:08.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T16:07:08.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:07:08.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T16:07:08.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:07:08.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:07:08.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:08.272 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:07:08.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:07:08.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:08.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:08.493 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:07:08.487+0000 7f89d3051540 -1 Falling back to public interface 2024-09-18T16:07:08.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:09.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:09.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:09.573 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:07:09.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:07:09.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:09.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:09.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:10.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:10.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:10.886 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:07:10.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:07:10.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:10.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:10.999 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:07:10.995+0000 7f89d3051540 -1 osd.3 0 log_to_monitors true 2024-09-18T16:07:11.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:12.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:12.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:12.319 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:07:12.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:07:12.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:12.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:12.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:13.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:13.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:13.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:07:13.643 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:07:13.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:13.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:14.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:15.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:15.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:15.031 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:07:15.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:07:15.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:15.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:15.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:16.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:16.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:16.349 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:07:16.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:07:16.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:16.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:16.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:17.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:17.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:17.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:07:17.664 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:07:17.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:17.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:18.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:19.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:19.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:19.027 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:07:19.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:07:19.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:19.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:19.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:20.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:20.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:20.350 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:07:20.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:07:20.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:20.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:20.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:21.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:21.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:21.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:07:21.666 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:07:21.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:21.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:22.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:23.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:23.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:23.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:07:23.023 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:07:23.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:23.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:23.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:24.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:24.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:24.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:07:24.645 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:07:24.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:24.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:25.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:26.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:26.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:26.013 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:07:26.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:07:26.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:26.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:26.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:27.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:27.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:27.824 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:07:27.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:07:27.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:27.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:28.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:29.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:29.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:29.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:07:29.567 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:07:29.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:29.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:30.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:31.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:31.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:31.105 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:07:31.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:07:31.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:31.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:31.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:32.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:32.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:32.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:07:32.505 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:07:32.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:32.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:32.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:33.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:33.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:33.828 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:07:33.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:07:33.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:33.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:34.196 INFO:tasks.workunit.client.0.smithi046.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/489712,v1:127.0.0.1:6827/489712] [v2:127.0.0.1:6828/489712,v1:127.0.0.1:6829/489712] exists,up 3b69bade-6a43-4474-9a9c-165b08807eaf 2024-09-18T16:07:34.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:07:34.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:07:34.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:07:34.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:876: unfound_erasure_coded: create_ec_pool ecpool true k=2 m=2 2024-09-18T16:07:34.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: create_ec_pool: local pool_name=ecpool 2024-09-18T16:07:34.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: create_ec_pool: shift 2024-09-18T16:07:34.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: create_ec_pool: local allow_overwrites=true 2024-09-18T16:07:34.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: create_ec_pool: shift 2024-09-18T16:07:34.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-09-18T16:07:34.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-18T16:07:34.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-18T16:07:35.544 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'ecpool' created 2024-09-18T16:07:35.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T16:07:36.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: create_ec_pool: '[' true = true ']' 2024-09-18T16:07:36.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-18T16:07:37.259 INFO:tasks.workunit.client.0.smithi046.stderr:set pool 1 allow_ec_overwrites to true 2024-09-18T16:07:37.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: create_ec_pool: wait_for_clean 2024-09-18T16:07:37.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:07:37.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:07:37.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:07:37.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:07:37.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:07:37.275 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:07:37.275 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:07:37.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:07:37.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:07:37.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:07:37.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:07:37.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:07:37.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:07:37.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:07:37.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:07:37.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:07:37.833 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:07:37.833 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T16:07:37.833 INFO:tasks.workunit.client.0.smithi046.stderr:3' 2024-09-18T16:07:37.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:07:37.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:07:37.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:07:38.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836505 2024-09-18T16:07:38.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836505 2024-09-18T16:07:38.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505' 2024-09-18T16:07:38.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:07:38.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:07:38.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672985 2024-09-18T16:07:38.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672985 2024-09-18T16:07:38.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-42949672985' 2024-09-18T16:07:38.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:07:38.152 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:07:38.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509456 2024-09-18T16:07:38.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509456 2024-09-18T16:07:38.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-42949672985 2-64424509456' 2024-09-18T16:07:38.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:07:38.297 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T16:07:38.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345926 2024-09-18T16:07:38.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345926 2024-09-18T16:07:38.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-42949672985 2-64424509456 3-85899345926' 2024-09-18T16:07:38.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:07:38.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836505 2024-09-18T16:07:38.436 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:07:38.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:07:38.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836505 2024-09-18T16:07:38.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:07:38.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836505 2024-09-18T16:07:38.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-18T16:07:38.443 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836505 2024-09-18T16:07:38.444 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:07:38.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836505 -lt 21474836505 2024-09-18T16:07:38.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:07:38.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672985 2024-09-18T16:07:38.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:07:38.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:07:38.754 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672985 2024-09-18T16:07:38.754 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:07:38.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672985 2024-09-18T16:07:38.757 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672985 2024-09-18T16:07:38.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-09-18T16:07:38.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:07:39.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672984 -lt 42949672985 2024-09-18T16:07:39.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:07:40.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:07:40.129 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:07:40.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672985 -lt 42949672985 2024-09-18T16:07:40.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:07:40.447 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509456 2024-09-18T16:07:40.447 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:07:40.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:07:40.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509456 2024-09-18T16:07:40.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:07:40.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509456 2024-09-18T16:07:40.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-09-18T16:07:40.453 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509456 2024-09-18T16:07:40.454 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:07:40.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509456 -lt 64424509456 2024-09-18T16:07:40.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:07:40.785 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345926 2024-09-18T16:07:40.785 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:07:40.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T16:07:40.790 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345926 2024-09-18T16:07:40.790 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:07:40.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345926 2024-09-18T16:07:40.792 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.3 seq 85899345926 2024-09-18T16:07:40.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345926' 2024-09-18T16:07:40.793 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T16:07:41.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345927 -lt 85899345926 2024-09-18T16:07:41.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:07:41.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:07:41.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:07:41.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:07:41.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:07:41.565 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:07:41.565 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:07:41.565 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:07:41.565 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:07:41.566 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:07:41.566 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:07:41.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:07:41.900 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:07:41.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:07:41.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:07:42.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:07:42.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:07:42.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:07:42.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: create_ec_pool: return 0 2024-09-18T16:07:42.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:878: unfound_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-18T16:07:42.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local dir=td/osd-scrub-repair 2024-09-18T16:07:42.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local poolname=ecpool 2024-09-18T16:07:42.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local obj=SOMETHING 2024-09-18T16:07:42.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local scrub=noscrub 2024-09-18T16:07:42.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: '[' noscrub = noscrub ']' 2024-09-18T16:07:42.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:77: add_something: ceph osd set noscrub 2024-09-18T16:07:42.752 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T16:07:42.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set nodeep-scrub 2024-09-18T16:07:43.307 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T16:07:43.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: local payload=ABCDEF 2024-09-18T16:07:43.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: echo ABCDEF 2024-09-18T16:07:43.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-18T16:07:43.572 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:880: unfound_erasure_coded: get_primary ecpool SOMETHING 2024-09-18T16:07:43.572 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-09-18T16:07:43.572 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T16:07:43.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-18T16:07:43.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T16:07:43.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:880: unfound_erasure_coded: local primary=1 2024-09-18T16:07:43.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:881: unfound_erasure_coded: osds=($(get_osds $poolname SOMETHING | sed -e "s/$primary//")) 2024-09-18T16:07:43.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:881: unfound_erasure_coded: get_osds ecpool SOMETHING 2024-09-18T16:07:43.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-09-18T16:07:43.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-18T16:07:43.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:881: unfound_erasure_coded: sed -e s/1// 2024-09-18T16:07:43.885 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-18T16:07:43.885 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T16:07:44.236 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-09-18T16:07:44.237 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T16:07:44.237 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T16:07:44.237 INFO:tasks.workunit.client.0.smithi046.stderr:3' 2024-09-18T16:07:44.237 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 3 2024-09-18T16:07:44.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:881: unfound_erasure_coded: local -a osds 2024-09-18T16:07:44.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:882: unfound_erasure_coded: local not_primary_first=0 2024-09-18T16:07:44.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:883: unfound_erasure_coded: local not_primary_second=2 2024-09-18T16:07:44.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:884: unfound_erasure_coded: local not_primary_third=3 2024-09-18T16:07:44.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:889: unfound_erasure_coded: pids= 2024-09-18T16:07:44.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-18T16:07:44.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T16:07:44.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T16:07:44.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 491274"' 2024-09-18T16:07:44.240 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 491274' 2024-09-18T16:07:44.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-18T16:07:44.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T16:07:44.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T16:07:44.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 491275"' 2024-09-18T16:07:44.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 491275' 2024-09-18T16:07:44.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:892: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T16:07:44.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: run_in_background: local pid_variable=pids 2024-09-18T16:07:44.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: run_in_background: shift 2024-09-18T16:07:44.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-18T16:07:44.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: eval 'pids+=" 491278"' 2024-09-18T16:07:44.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/491277: /' 2024-09-18T16:07:44.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: run_in_background: pids+=' 491278' 2024-09-18T16:07:44.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: unfound_erasure_coded: wait_background pids 2024-09-18T16:07:44.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: wait_background: pids=' 491274 491275 491278' 2024-09-18T16:07:44.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: wait_background: return_code=0 2024-09-18T16:07:44.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T16:07:44.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 491274 2024-09-18T16:07:44.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-18T16:07:44.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/491281: /' 2024-09-18T16:07:44.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T16:07:44.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: sed 's/^/491283: /' 2024-09-18T16:07:47.565 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T16:07:47.565 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T16:07:47.565 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-18T16:07:47.565 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T16:07:47.565 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-18T16:07:47.565 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T16:07:47.565 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T16:07:47.565 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-18T16:07:47.565 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T16:07:47.565 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-18T16:07:47.566 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:07:47.566 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:07:47.566 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:07:47.566 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:07:47.566 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:07:47.566 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:07:47.566 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-18T16:07:47.566 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T16:07:47.566 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T16:07:47.566 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T16:07:47.566 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T16:07:47.566 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-18T16:07:47.566 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-18T16:07:47.567 INFO:tasks.workunit.client.0.smithi046.stderr:491277: remove 1#1:eb822e21:::SOMETHING:head# 2024-09-18T16:07:47.567 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-18T16:07:47.567 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T16:07:47.567 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T16:07:47.567 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T16:07:47.567 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T16:07:47.567 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-18T16:07:47.567 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T16:07:47.567 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:07:47.572 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:07:47.572 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:07:47.572 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-18T16:07:47.572 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-18T16:07:47.572 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T16:07:47.572 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T16:07:47.572 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T16:07:47.573 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T16:07:47.573 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:07:47.573 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:07:47.573 INFO:tasks.workunit.client.0.smithi046.stderr:491277: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:07:47.573 INFO:tasks.workunit.client.0.smithi046.stderr:491277: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:07:47.573 INFO:tasks.workunit.client.0.smithi046.stderr:491277: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:07:47.573 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:07:47.573 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:07:47.573 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:07:47.573 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T16:07:47.573 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T16:07:47.573 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:07:47.573 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:07:47.573 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:07:47.574 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:07:47.574 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T16:07:47.574 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T16:07:47.574 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-18T16:07:47.574 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T16:07:47.574 INFO:tasks.workunit.client.0.smithi046.stderr:491277: start osd.0 2024-09-18T16:07:47.603 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T16:07:47.603 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T16:07:47.603 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-18T16:07:47.603 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T16:07:47.603 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T16:07:47.604 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T16:07:47.604 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T16:07:47.604 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-18T16:07:47.604 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T16:07:47.604 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-18T16:07:47.604 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:07:47.604 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:07:47.604 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:07:47.604 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:07:47.604 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:07:47.604 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:07:47.604 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-18T16:07:47.605 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T16:07:47.605 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T16:07:47.605 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-18T16:07:47.605 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T16:07:47.605 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-18T16:07:47.605 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-18T16:07:47.605 INFO:tasks.workunit.client.0.smithi046.stderr:491283: remove 3#1:eb822e21:::SOMETHING:head# 2024-09-18T16:07:47.605 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-18T16:07:47.605 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T16:07:47.605 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T16:07:47.605 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T16:07:47.605 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T16:07:47.605 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-18T16:07:47.606 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T16:07:47.606 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:07:47.608 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:07:47.608 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:07:47.608 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-18T16:07:47.608 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-18T16:07:47.609 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T16:07:47.609 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T16:07:47.609 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T16:07:47.609 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T16:07:47.609 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:07:47.609 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:07:47.609 INFO:tasks.workunit.client.0.smithi046.stderr:491283: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:07:47.609 INFO:tasks.workunit.client.0.smithi046.stderr:491283: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:07:47.609 INFO:tasks.workunit.client.0.smithi046.stderr:491283: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:07:47.609 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:07:47.609 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:07:47.609 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:07:47.610 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T16:07:47.610 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T16:07:47.610 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:07:47.610 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:07:47.610 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:07:47.610 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:07:47.610 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T16:07:47.610 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T16:07:47.610 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-18T16:07:47.610 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T16:07:47.610 INFO:tasks.workunit.client.0.smithi046.stderr:491283: start osd.3 2024-09-18T16:07:47.620 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-18T16:07:47.620 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T16:07:47.620 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-18T16:07:47.620 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T16:07:47.620 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-09-18T16:07:47.620 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-18T16:07:47.620 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T16:07:47.620 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-18T16:07:47.620 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T16:07:47.620 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-18T16:07:47.620 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:07:47.620 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:07:47.621 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:07:47.621 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:07:47.621 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:07:47.621 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:07:47.621 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-09-18T16:07:47.621 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-18T16:07:47.621 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T16:07:47.621 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-18T16:07:47.621 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T16:07:47.621 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-18T16:07:47.621 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-09-18T16:07:47.621 INFO:tasks.workunit.client.0.smithi046.stderr:491281: remove 2#1:eb822e21:::SOMETHING:head# 2024-09-18T16:07:47.621 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-18T16:07:47.622 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-09-18T16:07:47.622 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T16:07:47.622 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T16:07:47.622 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T16:07:47.622 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-18T16:07:47.622 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-18T16:07:47.622 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:07:47.627 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:07:47.627 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:07:47.627 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-18T16:07:47.627 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-18T16:07:47.627 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T16:07:47.627 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T16:07:47.627 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-18T16:07:47.627 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T16:07:47.627 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:07:47.627 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:07:47.627 INFO:tasks.workunit.client.0.smithi046.stderr:491281: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:07:47.627 INFO:tasks.workunit.client.0.smithi046.stderr:491281: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:07:47.627 INFO:tasks.workunit.client.0.smithi046.stderr:491281: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:07:47.628 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:07:47.628 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' 2024-09-18T16:07:47.628 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:07:47.628 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-18T16:07:47.628 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-18T16:07:47.628 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:07:47.628 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:07:47.628 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:07:47.628 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:07:47.628 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T16:07:47.628 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T16:07:47.628 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-18T16:07:47.629 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T16:07:47.629 INFO:tasks.workunit.client.0.smithi046.stderr:491281: start osd.2 2024-09-18T16:07:52.277 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=f9180b89-f6c0-4d8c-a01e-897a631bb4b8 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.137209/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:07:52.277 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-18T16:07:52.278 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T16:07:52.278 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T16:07:52.278 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T16:07:52.278 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T16:07:52.278 INFO:tasks.workunit.client.0.smithi046.stderr:491283: 2024-09-18T16:07:47.650+0000 7f3bfd79e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:07:52.278 INFO:tasks.workunit.client.0.smithi046.stderr:491283: 2024-09-18T16:07:47.654+0000 7f3bfd79e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:07:52.278 INFO:tasks.workunit.client.0.smithi046.stderr:491283: 2024-09-18T16:07:47.654+0000 7f3bfd79e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:07:52.278 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T16:07:52.278 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:07:52.278 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T16:07:52.278 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:07:52.278 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:07:52.278 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:52.279 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:07:52.279 INFO:tasks.workunit.client.0.smithi046.stderr:491283: 0 2024-09-18T16:07:52.279 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:52.279 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:52.279 INFO:tasks.workunit.client.0.smithi046.stderr:491283: 2024-09-18T16:07:47.970+0000 7f3bfd79e540 -1 Falling back to public interface 2024-09-18T16:07:52.279 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:52.279 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:52.279 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:52.279 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:07:52.279 INFO:tasks.workunit.client.0.smithi046.stderr:491283: 1 2024-09-18T16:07:52.279 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:52.279 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:52.279 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:52.279 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:52.280 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:52.280 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:07:52.280 INFO:tasks.workunit.client.0.smithi046.stderr:491283: 2 2024-09-18T16:07:52.280 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:52.280 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:52.280 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:52.280 INFO:tasks.workunit.client.0.smithi046.stderr:491283: 2024-09-18T16:07:51.646+0000 7f3bfd79e540 -1 osd.3 29 log_to_monitors true 2024-09-18T16:07:52.280 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:52.280 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:52.280 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:07:52.280 INFO:tasks.workunit.client.0.smithi046.stderr:491283: 3 2024-09-18T16:07:52.280 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:52.280 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:52.368 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-heops 2024-09-18T16:07:52.368 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-18T16:07:52.369 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T16:07:52.369 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T16:07:52.369 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T16:07:52.369 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T16:07:52.369 INFO:tasks.workunit.client.0.smithi046.stderr:491277: 2024-09-18T16:07:47.598+0000 7f3bf7d64540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:07:52.369 INFO:tasks.workunit.client.0.smithi046.stderr:491277: 2024-09-18T16:07:47.602+0000 7f3bf7d64540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:07:52.369 INFO:tasks.workunit.client.0.smithi046.stderr:491277: 2024-09-18T16:07:47.602+0000 7f3bf7d64540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:07:52.369 INFO:tasks.workunit.client.0.smithi046.stderr:491277: 2024-09-18T16:07:47.910+0000 7f3bf7d64540 -1 Falling back to public interface 2024-09-18T16:07:52.369 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T16:07:52.369 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:07:52.369 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:07:52.369 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:07:52.369 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:07:52.369 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:52.370 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:07:52.370 INFO:tasks.workunit.client.0.smithi046.stderr:491277: 0 2024-09-18T16:07:52.370 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:07:52.370 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:52.370 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:52.370 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:52.370 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:52.370 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:07:52.370 INFO:tasks.workunit.client.0.smithi046.stderr:491277: 1 2024-09-18T16:07:52.370 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:52.370 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:07:52.370 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:52.370 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:52.371 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:52.371 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:07:52.371 INFO:tasks.workunit.client.0.smithi046.stderr:491277: 2 2024-09-18T16:07:52.371 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:52.371 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:07:52.371 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:52.371 INFO:tasks.workunit.client.0.smithi046.stderr:491277: 2024-09-18T16:07:51.454+0000 7f3bf7d64540 -1 osd.0 29 log_to_monitors true 2024-09-18T16:07:52.371 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:52.371 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:52.371 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:07:52.371 INFO:tasks.workunit.client.0.smithi046.stderr:491277: 3 2024-09-18T16:07:52.371 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:52.371 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:07:52.512 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-heops 2024-09-18T16:07:52.512 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-18T16:07:52.513 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T16:07:52.513 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T16:07:52.513 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T16:07:52.513 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T16:07:52.513 INFO:tasks.workunit.client.0.smithi046.stderr:491281: 2024-09-18T16:07:47.658+0000 7f2bf9385540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:07:52.513 INFO:tasks.workunit.client.0.smithi046.stderr:491281: 2024-09-18T16:07:47.662+0000 7f2bf9385540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:07:52.513 INFO:tasks.workunit.client.0.smithi046.stderr:491281: 2024-09-18T16:07:47.666+0000 7f2bf9385540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:07:52.513 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T16:07:52.513 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:07:52.513 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T16:07:52.513 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:07:52.513 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:07:52.513 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:52.514 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:07:52.514 INFO:tasks.workunit.client.0.smithi046.stderr:491281: 0 2024-09-18T16:07:52.514 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:52.514 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:07:52.514 INFO:tasks.workunit.client.0.smithi046.stderr:491281: 2024-09-18T16:07:47.982+0000 7f2bf9385540 -1 Falling back to public interface 2024-09-18T16:07:52.514 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:52.514 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:52.514 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:52.514 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:07:52.514 INFO:tasks.workunit.client.0.smithi046.stderr:491281: 1 2024-09-18T16:07:52.514 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:52.514 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:07:52.514 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:52.515 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:52.515 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:52.515 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:07:52.515 INFO:tasks.workunit.client.0.smithi046.stderr:491281: 2 2024-09-18T16:07:52.515 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:52.515 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:07:52.515 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:52.515 INFO:tasks.workunit.client.0.smithi046.stderr:491281: 2024-09-18T16:07:51.038+0000 7f2bf9385540 -1 osd.2 29 log_to_monitors true 2024-09-18T16:07:52.515 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:52.515 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:52.515 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:07:52.515 INFO:tasks.workunit.client.0.smithi046.stderr:491281: 3 2024-09-18T16:07:52.515 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:07:52.515 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:54.404 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:54.404 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:54.405 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:54.405 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:07:54.405 INFO:tasks.workunit.client.0.smithi046.stderr:491283: 4 2024-09-18T16:07:54.405 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:54.405 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:54.405 INFO:tasks.workunit.client.0.smithi046.stderr:491283: osd.3 up in weight 1 up_from 35 up_thru 0 down_at 30 last_clean_interval [20,29) [v2:127.0.0.1:6818/492515,v1:127.0.0.1:6819/492515] [v2:127.0.0.1:6820/492515,v1:127.0.0.1:6821/492515] exists,up 3b69bade-6a43-4474-9a9c-165b08807eaf 2024-09-18T16:07:54.405 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:07:54.405 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:07:54.405 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:07:54.405 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T16:07:54.405 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:07:54.405 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:07:54.406 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:07:54.406 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:07:54.406 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:07:54.406 INFO:tasks.workunit.client.0.smithi046.stderr:491283: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:07:54.406 INFO:tasks.workunit.client.0.smithi046.stderr:491283: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:07:54.406 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:07:54.406 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:07:54.406 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:07:54.406 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:07:54.406 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:07:54.406 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:07:54.406 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:07:54.407 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:07:54.407 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:07:54.407 INFO:tasks.workunit.client.0.smithi046.stderr:491283: 1 2024-09-18T16:07:54.407 INFO:tasks.workunit.client.0.smithi046.stderr:491283: 2 2024-09-18T16:07:54.407 INFO:tasks.workunit.client.0.smithi046.stderr:491283: 3' 2024-09-18T16:07:54.407 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:07:54.407 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:07:54.407 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:07:54.407 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855362 2024-09-18T16:07:54.407 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855362 2024-09-18T16:07:54.407 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855362' 2024-09-18T16:07:54.407 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:07:54.408 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph lpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:54.408 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:54.408 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:54.408 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:07:54.408 INFO:tasks.workunit.client.0.smithi046.stderr:491277: 4 2024-09-18T16:07:54.408 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:54.408 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:07:54.408 INFO:tasks.workunit.client.0.smithi046.stderr:491277: osd.0 up in weight 1 up_from 35 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/492486,v1:127.0.0.1:6803/492486] [v2:127.0.0.1:6804/492486,v1:127.0.0.1:6805/492486] exists,up 773229d1-e50b-4e96-a576-954bbcfa4f77 2024-09-18T16:07:54.408 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:07:54.408 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:07:54.408 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:07:54.408 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T16:07:54.409 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:07:54.409 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:07:54.409 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:07:54.409 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:07:54.409 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:07:54.409 INFO:tasks.workunit.client.0.smithi046.stderr:491277: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:07:54.409 INFO:tasks.workunit.client.0.smithi046.stderr:491277: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:07:54.409 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:07:54.409 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:07:54.409 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:07:54.409 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:07:54.409 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:07:54.409 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:07:54.410 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:07:54.410 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:07:54.410 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:07:54.410 INFO:tasks.workunit.client.0.smithi046.stderr:491277: 1 2024-09-18T16:07:54.410 INFO:tasks.workunit.client.0.smithi046.stderr:491277: 2 2024-09-18T16:07:54.410 INFO:tasks.workunit.client.0.smithi046.stderr:491277: 3' 2024-09-18T16:07:54.410 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:07:54.410 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:07:54.410 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:07:54.410 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855363 2024-09-18T16:07:54.410 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855363 2024-09-18T16:07:54.410 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855363' 2024-09-18T16:07:54.410 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:07:54.551 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tlpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:54.551 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:54.551 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:54.551 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:07:54.551 INFO:tasks.workunit.client.0.smithi046.stderr:491281: 4 2024-09-18T16:07:54.552 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:54.552 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:07:54.552 INFO:tasks.workunit.client.0.smithi046.stderr:491281: osd.2 up in weight 1 up_from 35 up_thru 0 down_at 30 last_clean_interval [15,29) [v2:127.0.0.1:6826/492532,v1:127.0.0.1:6827/492532] [v2:127.0.0.1:6828/492532,v1:127.0.0.1:6829/492532] exists,up a2f23771-2d72-49c9-b0d1-4d58d31d6f51 2024-09-18T16:07:54.552 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:07:54.552 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:07:54.552 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:07:54.552 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T16:07:54.552 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:07:54.552 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:07:54.552 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:07:54.552 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:07:54.552 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:07:54.553 INFO:tasks.workunit.client.0.smithi046.stderr:491281: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:07:54.553 INFO:tasks.workunit.client.0.smithi046.stderr:491281: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:07:54.553 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:07:54.553 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:07:54.553 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:07:54.553 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:07:54.553 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:07:54.553 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:07:54.553 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:07:54.553 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:07:54.553 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:07:54.553 INFO:tasks.workunit.client.0.smithi046.stderr:491281: 1 2024-09-18T16:07:54.553 INFO:tasks.workunit.client.0.smithi046.stderr:491281: 2 2024-09-18T16:07:54.554 INFO:tasks.workunit.client.0.smithi046.stderr:491281: 3' 2024-09-18T16:07:54.554 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:07:54.554 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:07:54.554 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:07:54.554 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855364 2024-09-18T16:07:54.554 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855364 2024-09-18T16:07:54.554 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855364' 2024-09-18T16:07:54.554 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:07:56.437 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:07:56.437 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672989 2024-09-18T16:07:56.437 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672989 2024-09-18T16:07:56.437 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855362 1-42949672989' 2024-09-18T16:07:56.437 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:07:56.437 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:07:56.438 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855363 2024-09-18T16:07:56.438 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855363 2024-09-18T16:07:56.438 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855362 1-42949672989 2-150323855363' 2024-09-18T16:07:56.438 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:07:56.438 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T16:07:56.438 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855363 2024-09-18T16:07:56.438 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855363 2024-09-18T16:07:56.438 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855362 1-42949672989 2-150323855363 3-150323855363' 2024-09-18T16:07:56.438 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:07:56.438 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-150323855362 2024-09-18T16:07:56.438 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:07:56.438 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:07:56.439 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-150323855362 2024-09-18T16:07:56.439 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:07:56.439 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855362 2024-09-18T16:07:56.439 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-09-18T16:07:56.439 INFO:tasks.workunit.client.0.smithi046.stderr:491283: waiting osd.0 seq 150323855362 2024-09-18T16:07:56.439 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:07:56.439 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 150323855362 2024-09-18T16:07:56.439 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:07:56.439 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:07:56.439 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:07:56.439 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855364 -lt 150323855362 2024-09-18T16:07:56.439 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:07:56.439 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672989 2024-09-18T16:07:56.440 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:07:56.440 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:07:56.440 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672989 2024-09-18T16:07:56.440 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:07:56.451 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalonell osd.1 flush_pg_stats 2024-09-18T16:07:56.452 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672990 2024-09-18T16:07:56.452 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672990 2024-09-18T16:07:56.452 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855363 1-42949672990' 2024-09-18T16:07:56.452 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:07:56.452 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:07:56.452 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855362 2024-09-18T16:07:56.452 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855362 2024-09-18T16:07:56.452 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855363 1-42949672990 2-150323855362' 2024-09-18T16:07:56.452 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:07:56.452 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T16:07:56.452 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855362 2024-09-18T16:07:56.452 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855362 2024-09-18T16:07:56.453 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855363 1-42949672990 2-150323855362 3-150323855362' 2024-09-18T16:07:56.453 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:07:56.453 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-150323855363 2024-09-18T16:07:56.453 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:07:56.453 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:07:56.453 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-150323855363 2024-09-18T16:07:56.453 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:07:56.453 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855363 2024-09-18T16:07:56.453 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 150323855363' 2024-09-18T16:07:56.453 INFO:tasks.workunit.client.0.smithi046.stderr:491277: waiting osd.0 seq 150323855363 2024-09-18T16:07:56.453 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:07:56.453 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 150323855363 2024-09-18T16:07:56.453 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:07:56.454 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:07:56.454 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:07:56.454 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855364 -lt 150323855363 2024-09-18T16:07:56.454 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:07:56.454 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672990 2024-09-18T16:07:56.454 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:07:56.454 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:07:56.454 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672990 2024-09-18T16:07:56.454 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:07:56.590 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalonetell osd.1 flush_pg_stats 2024-09-18T16:07:56.590 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672991 2024-09-18T16:07:56.590 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672991 2024-09-18T16:07:56.590 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855364 1-42949672991' 2024-09-18T16:07:56.591 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:07:56.591 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:07:56.591 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855364 2024-09-18T16:07:56.591 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855364 2024-09-18T16:07:56.591 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855364 1-42949672991 2-150323855364' 2024-09-18T16:07:56.591 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:07:56.591 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T16:07:56.591 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855364 2024-09-18T16:07:56.591 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855364 2024-09-18T16:07:56.591 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-150323855364 1-42949672991 2-150323855364 3-150323855364' 2024-09-18T16:07:56.591 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:07:56.591 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-150323855364 2024-09-18T16:07:56.591 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:07:56.592 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:07:56.592 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-150323855364 2024-09-18T16:07:56.592 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:07:56.592 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855364 2024-09-18T16:07:56.592 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 150323855364' 2024-09-18T16:07:56.592 INFO:tasks.workunit.client.0.smithi046.stderr:491281: waiting osd.0 seq 150323855364 2024-09-18T16:07:56.592 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:07:56.592 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 150323855364 2024-09-18T16:07:56.592 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:07:56.592 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:07:56.592 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:07:56.592 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855364 -lt 150323855364 2024-09-18T16:07:56.592 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:07:56.593 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672991 2024-09-18T16:07:56.593 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:07:56.593 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:07:56.593 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672991 2024-09-18T16:07:56.593 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:07:57.883 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672989 2024-09-18T16:07:57.884 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-09-18T16:07:57.884 INFO:tasks.workunit.client.0.smithi046.stderr:491283: waiting osd.1 seq 42949672989 2024-09-18T16:07:57.884 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:07:57.884 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672992 -lt 42949672989 2024-09-18T16:07:57.884 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:07:57.884 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:07:57.884 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-150323855363 2024-09-18T16:07:57.884 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:07:57.884 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-150323855363 2024-09-18T16:07:57.884 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:07:57.884 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855363 2024-09-18T16:07:57.884 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 150323855363' 2024-09-18T16:07:57.885 INFO:tasks.workunit.client.0.smithi046.stderr:491283: waiting osd.2 seq 150323855363 2024-09-18T16:07:57.885 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:07:57.885 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855364 -lt 150323855363 2024-09-18T16:07:57.885 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:07:57.885 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-150323855363 2024-09-18T16:07:57.885 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:07:57.885 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T16:07:57.885 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-150323855363 2024-09-18T16:07:57.885 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:07:57.885 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855363 2024-09-18T16:07:57.885 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 150323855363' 2024-09-18T16:07:57.885 INFO:tasks.workunit.client.0.smithi046.stderr:491283: waiting osd.3 seq 150323855363 2024-09-18T16:07:57.886 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T16:07:57.886 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855364 -lt 150323855363 2024-09-18T16:07:57.886 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:07:57.886 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:07:57.886 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:07:57.886 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:07:57.886 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:07:57.886 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:07:57.886 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:07:57.886 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:07:57.886 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:07:57.968 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest//ceph-helpers.sh:2183: flush_pg_stats: seq=42949672990 2024-09-18T16:07:57.968 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672990' 2024-09-18T16:07:57.968 INFO:tasks.workunit.client.0.smithi046.stderr:491277: waiting osd.1 seq 42949672990 2024-09-18T16:07:57.968 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:07:57.968 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672992 -lt 42949672990 2024-09-18T16:07:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:07:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:07:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-150323855362 2024-09-18T16:07:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:07:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-150323855362 2024-09-18T16:07:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:07:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855362 2024-09-18T16:07:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 150323855362' 2024-09-18T16:07:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:491277: waiting osd.2 seq 150323855362 2024-09-18T16:07:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:07:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855364 -lt 150323855362 2024-09-18T16:07:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:07:57.969 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-150323855362 2024-09-18T16:07:57.970 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:07:57.970 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T16:07:57.970 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-150323855362 2024-09-18T16:07:57.970 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:07:57.970 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855362 2024-09-18T16:07:57.970 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 150323855362' 2024-09-18T16:07:57.970 INFO:tasks.workunit.client.0.smithi046.stderr:491277: waiting osd.3 seq 150323855362 2024-09-18T16:07:57.970 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T16:07:57.970 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855364 -lt 150323855362 2024-09-18T16:07:57.970 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:07:57.970 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:07:57.970 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:07:57.970 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:07:57.971 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:07:57.971 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:07:57.971 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:07:57.971 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:07:57.971 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:07:58.030 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/ce/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672991 2024-09-18T16:07:58.031 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672991' 2024-09-18T16:07:58.031 INFO:tasks.workunit.client.0.smithi046.stderr:491281: waiting osd.1 seq 42949672991 2024-09-18T16:07:58.031 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:07:58.031 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672992 -lt 42949672991 2024-09-18T16:07:58.031 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:07:58.031 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-150323855364 2024-09-18T16:07:58.031 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:07:58.031 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:07:58.031 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-150323855364 2024-09-18T16:07:58.031 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:07:58.031 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855364 2024-09-18T16:07:58.031 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 150323855364' 2024-09-18T16:07:58.032 INFO:tasks.workunit.client.0.smithi046.stderr:491281: waiting osd.2 seq 150323855364 2024-09-18T16:07:58.032 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:07:58.032 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855364 -lt 150323855364 2024-09-18T16:07:58.032 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:07:58.032 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-150323855364 2024-09-18T16:07:58.032 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:07:58.032 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T16:07:58.032 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:07:58.032 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-150323855364 2024-09-18T16:07:58.032 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855364 2024-09-18T16:07:58.032 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 150323855364' 2024-09-18T16:07:58.032 INFO:tasks.workunit.client.0.smithi046.stderr:491281: waiting osd.3 seq 150323855364 2024-09-18T16:07:58.032 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T16:07:58.032 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855364 -lt 150323855364 2024-09-18T16:07:58.033 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:07:58.033 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:07:58.033 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:07:58.033 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:07:58.033 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:07:58.033 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:07:58.033 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:07:58.033 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:07:58.033 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:07:58.603 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:07:58.603 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:07:58.604 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:07:58.604 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:07:58.604 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:07:58.604 INFO:tasks.workunit.client.0.smithi046.stderr:491283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:07:58.604 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:07:58.604 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:07:58.604 INFO:tasks.workunit.client.0.smithi046.stderr:491283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:07:58.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T16:07:58.676 INFO:tasks.workunit.client.0.smithi046.stderr:lone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:07:58.676 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:07:58.676 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:07:58.676 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:07:58.676 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:07:58.676 INFO:tasks.workunit.client.0.smithi046.stderr:491277: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:07:58.676 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:07:58.676 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:07:58.676 INFO:tasks.workunit.client.0.smithi046.stderr:491277: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:07:58.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T16:07:58.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T16:07:58.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 491275 2024-09-18T16:07:58.766 INFO:tasks.workunit.client.0.smithi046.stderr:clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:07:58.766 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:07:58.766 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:07:58.766 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:07:58.767 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:07:58.767 INFO:tasks.workunit.client.0.smithi046.stderr:491281: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:07:58.767 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:07:58.767 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:07:58.767 INFO:tasks.workunit.client.0.smithi046.stderr:491281: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:07:58.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: run_in_background: return 0 2024-09-18T16:07:58.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: wait_background: for pid in $pids 2024-09-18T16:07:58.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: wait_background: wait 491278 2024-09-18T16:07:58.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: eval 'pids='\'''\''' 2024-09-18T16:07:58.769 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: wait_background: pids= 2024-09-18T16:07:58.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: wait_background: return 0 2024-09-18T16:07:58.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: return_code=0 2024-09-18T16:07:58.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:895: unfound_erasure_coded: '[' 0 -ne 0 ']' 2024-09-18T16:07:58.769 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:900: unfound_erasure_coded: get_pg ecpool SOMETHING 2024-09-18T16:07:58.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-09-18T16:07:58.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T16:07:58.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-18T16:07:58.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T16:07:59.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:900: unfound_erasure_coded: local pg=1.0 2024-09-18T16:07:59.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:901: unfound_erasure_coded: repair 1.0 2024-09-18T16:07:59.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=1.0 2024-09-18T16:07:59.100 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 1.0 2024-09-18T16:07:59.100 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:07:59.100 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:07:59.101 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:07:59.101 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:07:59.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T16:07:35.348546+0000 2024-09-18T16:07:59.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 1.0 2024-09-18T16:07:59.689 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-09-18T16:07:59.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 1.0 2024-09-18T16:07:35.348546+0000 2024-09-18T16:07:59.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-18T16:07:59.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T16:07:35.348546+0000 2024-09-18T16:07:59.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T16:07:59.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T16:07:59.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:07:59.699 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:07:59.699 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:07:59.699 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:07:59.699 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:07:59.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:07:59.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:07:35.348546+0000 '>' 2024-09-18T16:07:35.348546+0000 2024-09-18T16:07:59.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:08:01.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:08:01.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:08:01.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:08:01.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:08:01.003 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:08:01.003 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:08:01.003 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:08:01.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:07:35.348546+0000 '>' 2024-09-18T16:07:35.348546+0000 2024-09-18T16:08:01.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:08:02.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:08:02.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:08:02.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:08:02.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:08:02.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:08:02.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:08:02.327 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:08:02.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:07:35.348546+0000 '>' 2024-09-18T16:07:35.348546+0000 2024-09-18T16:08:02.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:08:03.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:08:03.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:08:03.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:08:03.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:08:03.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:08:03.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:08:03.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:08:03.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:07:35.348546+0000 '>' 2024-09-18T16:07:35.348546+0000 2024-09-18T16:08:03.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:08:04.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:08:04.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:08:04.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:08:04.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:08:04.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:08:04.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:08:04.956 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:08:05.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:07:35.348546+0000 '>' 2024-09-18T16:07:35.348546+0000 2024-09-18T16:08:05.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:08:06.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:08:06.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:08:06.260 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:08:06.260 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:08:06.260 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:08:06.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:08:06.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:08:06.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:08:01.107337+0000 '>' 2024-09-18T16:07:35.348546+0000 2024-09-18T16:08:06.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T16:08:06.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:906: unfound_erasure_coded: seq 1 60 2024-09-18T16:08:06.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:906: unfound_erasure_coded: for f in `seq 1 60` 2024-09-18T16:08:06.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:907: unfound_erasure_coded: ceph -s 2024-09-18T16:08:06.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:907: unfound_erasure_coded: grep '1/1 objects unfound' 2024-09-18T16:08:06.928 INFO:tasks.workunit.client.0.smithi046.stdout: 1/1 objects unfound (100.000%) 2024-09-18T16:08:06.928 INFO:tasks.workunit.client.0.smithi046.stdout: 1/1 objects unfound (100.000%) 2024-09-18T16:08:06.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:907: unfound_erasure_coded: break 2024-09-18T16:08:06.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:910: unfound_erasure_coded: ceph -s 2024-09-18T16:08:06.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:910: unfound_erasure_coded: grep '4 up' 2024-09-18T16:08:07.295 INFO:tasks.workunit.client.0.smithi046.stdout: osd: 4 osds: 4 up (since 14s), 4 in (since 66s) 2024-09-18T16:08:07.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:911: unfound_erasure_coded: ceph -s 2024-09-18T16:08:07.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:911: unfound_erasure_coded: grep '4 in' 2024-09-18T16:08:07.704 INFO:tasks.workunit.client.0.smithi046.stdout: osd: 4 osds: 4 up (since 15s), 4 in (since 66s) 2024-09-18T16:08:07.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:912: unfound_erasure_coded: ceph -s 2024-09-18T16:08:07.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:912: unfound_erasure_coded: grep '1/1 objects unfound' 2024-09-18T16:08:08.060 INFO:tasks.workunit.client.0.smithi046.stdout: 1/1 objects unfound (100.000%) 2024-09-18T16:08:08.060 INFO:tasks.workunit.client.0.smithi046.stdout: 1/1 objects unfound (100.000%) 2024-09-18T16:08:08.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: teardown td/osd-scrub-repair 2024-09-18T16:08:08.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T16:08:08.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:08:08.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T16:08:08.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:08:08.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:08:08.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:08:08.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:08:08.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:08:08.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:08:08.187 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:08:08.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:08:08.190 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:08:08.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:08:08.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:08:08.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:08:08.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:08:08.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:08:08.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:08:08.197 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:08:08.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:08:08.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:08:08.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T16:08:08.223 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:08:08.223 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:08:08.224 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:08:08.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T16:08:08.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:08:08.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:08:08.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T16:08:08.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-scrub-repair 0 2024-09-18T16:08:08.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-09-18T16:08:08.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T16:08:08.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-18T16:08:08.227 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:08:08.227 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:08:08.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:08:08.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:08:08.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:08:08.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:08:08.234 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:08:08.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:08:08.237 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:08:08.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:08:08.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:08:08.240 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:08:08.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:08:08.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:08:08.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:08:08.244 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:08:08.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:08:08.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T16:08:08.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-09-18T16:08:08.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:08:08.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:08:08.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137209 2024-09-18T16:08:08.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137209 2024-09-18T16:08:08.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:08:08.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:08:08.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T16:08:08.254 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T16:08:08.255 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T16:08:08.269 INFO:tasks.workunit:Running workunit scrub/osd-scrub-snaps.sh... 2024-09-18T16:08:08.269 DEBUG:teuthology.orchestra.run.smithi046:workunit test scrub/osd-scrub-snaps.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh 2024-09-18T16:08:08.286 INFO:tasks.workunit.client.0.smithi046.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T16:08:08.301 INFO:tasks.workunit.client.0.smithi046.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T16:08:08.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-18T16:08:08.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-18T16:08:08.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T16:08:08.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T16:08:08.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T16:08:08.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T16:08:08.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T16:08:08.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T16:08:08.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-scrub-snaps 2024-09-18T16:08:08.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:27: run: local dir=td/osd-scrub-snaps 2024-09-18T16:08:08.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:28: run: shift 2024-09-18T16:08:08.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:30: run: export CEPH_MON=127.0.0.1:7121 2024-09-18T16:08:08.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:30: run: CEPH_MON=127.0.0.1:7121 2024-09-18T16:08:08.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:31: run: export CEPH_ARGS 2024-09-18T16:08:08.302 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:32: run: uuidgen 2024-09-18T16:08:08.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:32: run: CEPH_ARGS+='--fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none ' 2024-09-18T16:08:08.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:33: run: CEPH_ARGS+='--mon-host=127.0.0.1:7121 ' 2024-09-18T16:08:08.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:35: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-09-18T16:08:08.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: set 2024-09-18T16:08:08.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T16:08:08.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: local 'funcs=TEST_scrub_snaps 2024-09-18T16:08:08.310 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_scrub_snaps_primary 2024-09-18T16:08:08.310 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_scrub_snaps_replica' 2024-09-18T16:08:08.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2024-09-18T16:08:08.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2024-09-18T16:08:08.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-snaps 2024-09-18T16:08:08.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-snaps 2024-09-18T16:08:08.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-09-18T16:08:08.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:08:08.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-18T16:08:08.311 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:08:08.311 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:08:08.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:08:08.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:08:08.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:08:08.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:08:08.315 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:08:08.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:08:08.317 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:08:08.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:08:08.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:08:08.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:08:08.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:08:08.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:08:08.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:08:08.322 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:08:08.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:08:08.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:08:08.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-09-18T16:08:08.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:08:08.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:08:08.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:08:08.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.495962 2024-09-18T16:08:08.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:08:08.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:08:08.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-snaps 2024-09-18T16:08:08.330 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T16:08:08.330 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:08:08.330 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:08:08.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.495962 2024-09-18T16:08:08.333 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T16:08:08.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T16:08:08.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T16:08:08.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T16:08:08.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2024-09-18T16:08:08.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps td/osd-scrub-snaps 2024-09-18T16:08:08.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:171: TEST_scrub_snaps: local dir=td/osd-scrub-snaps 2024-09-18T16:08:08.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:172: TEST_scrub_snaps: local poolname=test 2024-09-18T16:08:08.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:173: TEST_scrub_snaps: local OBJS=16 2024-09-18T16:08:08.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:174: TEST_scrub_snaps: local OSDS=1 2024-09-18T16:08:08.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:176: TEST_scrub_snaps: TESTDATA=testdata.495962 2024-09-18T16:08:08.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:178: TEST_scrub_snaps: run_mon td/osd-scrub-snaps a --osd_pool_default_size=1 2024-09-18T16:08:08.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-snaps 2024-09-18T16:08:08.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T16:08:08.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T16:08:08.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T16:08:08.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-snaps/a 2024-09-18T16:08:08.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=1 2024-09-18T16:08:08.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T16:08:08.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:08:08.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:08:08.709 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:08:08.709 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:08:08.709 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:08:08.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.495962/$cluster-$name.asok' 2024-09-18T16:08:08.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 2024-09-18T16:08:08.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T16:08:08.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T16:08:08.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:08:08.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:08:08.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T16:08:08.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T16:08:08.982 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:08:08.982 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:08:08.982 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:08:08.987 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:08:08.987 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:08:08.987 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:08:08.988 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.495962/ceph-mon.a.asok 2024-09-18T16:08:08.988 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:08:08.988 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.495962/ceph-mon.a.asok config get fsid 2024-09-18T16:08:09.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T16:08:09.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:08:09.083 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:08:09.083 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T16:08:09.083 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T16:08:09.084 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:08:09.084 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:08:09.084 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:08:09.084 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:08:09.084 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:08:09.084 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:08:09.085 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.495962/ceph-mon.a.asok 2024-09-18T16:08:09.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:08:09.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.495962/ceph-mon.a.asok config get mon_host 2024-09-18T16:08:09.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:179: TEST_scrub_snaps: run_mgr td/osd-scrub-snaps x 2024-09-18T16:08:09.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-snaps 2024-09-18T16:08:09.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T16:08:09.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T16:08:09.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T16:08:09.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-snaps/x 2024-09-18T16:08:09.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T16:08:10.025 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T16:08:10.025 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:08:10.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:08:10.026 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:08:10.026 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:08:10.026 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:08:10.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.495962/$cluster-$name.asok' 2024-09-18T16:08:10.027 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:08:10.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:08:10.064 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: expr 1 - 1 2024-09-18T16:08:10.065 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: seq 0 0 2024-09-18T16:08:10.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:08:10.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:182: TEST_scrub_snaps: run_osd td/osd-scrub-snaps 0 2024-09-18T16:08:10.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-snaps 2024-09-18T16:08:10.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:08:10.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T16:08:10.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:08:10.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-snaps/0 2024-09-18T16:08:10.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-09-18T16:08:10.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:08:10.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:08:10.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:08:10.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-09-18T16:08:10.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-09-18T16:08:10.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:08:10.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:08:10.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-18T16:08:10.069 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:08:10.069 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:08:10.069 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:08:10.069 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:08:10.069 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:08:10.069 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:08:10.069 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.495962/$cluster-$name.asok' 2024-09-18T16:08:10.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' 2024-09-18T16:08:10.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:08:10.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:08:10.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:08:10.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-18T16:08:10.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-18T16:08:10.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:08:10.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:08:10.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:08:10.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:08:10.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:08:10.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:08:10.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-snaps/0 2024-09-18T16:08:10.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:08:10.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d53e95df-af76-4ea0-a7bf-cb6c49f50b24 2024-09-18T16:08:10.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d53e95df-af76-4ea0-a7bf-cb6c49f50b24' 2024-09-18T16:08:10.072 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 d53e95df-af76-4ea0-a7bf-cb6c49f50b24 2024-09-18T16:08:10.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:08:10.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDq+upmGS8gBRAAB5o67cU6nynf1e329N9iDA== 2024-09-18T16:08:10.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDq+upmGS8gBRAAB5o67cU6nynf1e329N9iDA=="}' 2024-09-18T16:08:10.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d53e95df-af76-4ea0-a7bf-cb6c49f50b24 -i td/osd-scrub-snaps/0/new.json 2024-09-18T16:08:10.382 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:08:10.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-snaps/0/new.json 2024-09-18T16:08:10.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDq+upmGS8gBRAAB5o67cU6nynf1e329N9iDA== --osd-uuid d53e95df-af76-4ea0-a7bf-cb6c49f50b24 2024-09-18T16:08:10.421 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:08:10.414+0000 7f8ac0287540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:08:10.424 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:08:10.418+0000 7f8ac0287540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:08:10.426 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:08:10.422+0000 7f8ac0287540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:08:10.427 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:08:10.422+0000 7f8ac0287540 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-09-18T16:08:10.427 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:08:10.422+0000 7f8ac0287540 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-09-18T16:08:10.427 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:08:10.422+0000 7f8ac0287540 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-09-18T16:08:10.506 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:08:10.502+0000 7f8ac0287540 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2024-09-18T16:08:16.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2024-09-18T16:08:16.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:08:16.702 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T16:08:16.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T16:08:16.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:08:17.075 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T16:08:17.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T16:08:17.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:08:17.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:08:17.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:08:17.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:08:17.103 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:08:17.098+0000 7f7192a3e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:08:17.106 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:08:17.098+0000 7f7192a3e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:08:17.107 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:08:17.102+0000 7f7192a3e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:08:17.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T16:08:17.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:08:17.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:08:17.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:08:17.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:08:17.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:17.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:08:17.416 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:08:17.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:17.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:08:17.649 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:08:17.642+0000 7f7192a3e540 -1 Falling back to public interface 2024-09-18T16:08:17.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:18.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:18.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:18.721 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:08:18.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:08:18.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:18.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:08:19.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:20.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:20.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:20.032 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:08:20.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:08:20.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:20.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:08:20.135 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:08:20.130+0000 7f7192a3e540 -1 osd.0 0 log_to_monitors true 2024-09-18T16:08:20.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:21.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:21.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:21.396 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:08:21.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:08:21.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:21.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:08:21.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:22.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:22.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:22.813 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:08:22.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:08:22.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:22.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:08:23.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:24.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:24.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:24.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:08:24.133 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:08:24.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:24.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:08:24.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:25.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:25.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:25.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:08:25.442 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:08:25.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:25.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:08:25.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:26.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:26.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:26.838 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:08:26.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:08:26.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:26.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:08:27.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:28.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:28.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:28.148 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:08:28.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:08:28.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:28.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:08:28.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:29.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:29.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:29.464 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:08:29.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:08:29.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:29.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:08:29.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:30.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:30.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:30.831 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:08:30.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:08:30.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:30.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:08:31.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:32.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:32.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:32.142 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:08:32.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:08:32.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:32.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:08:32.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:33.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:33.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:33.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:08:33.449 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:08:33.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:33.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:08:34.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:35.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:35.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:35.042 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:08:35.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:08:35.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:35.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:08:35.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:36.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:36.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:36.889 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:08:36.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:08:36.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:36.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:08:37.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:38.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:38.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:38.508 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:08:38.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:08:38.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:38.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:08:39.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:40.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:40.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:40.185 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:08:40.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:08:40.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:40.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:08:40.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:41.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:41.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:41.531 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:08:41.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:08:41.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:41.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:08:41.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:42.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:42.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:42.973 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:08:42.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:08:42.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:42.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:08:43.278 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/497082,v1:127.0.0.1:6803/497082] [v2:127.0.0.1:6804/497082,v1:127.0.0.1:6805/497082] exists,up d53e95df-af76-4ea0-a7bf-cb6c49f50b24 2024-09-18T16:08:43.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:08:43.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:08:43.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:08:43.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:186: TEST_scrub_snaps: ceph osd set noscrub 2024-09-18T16:08:43.779 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T16:08:43.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:187: TEST_scrub_snaps: ceph osd set nodeep-scrub 2024-09-18T16:08:44.323 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T16:08:44.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:190: TEST_scrub_snaps: create_pool test 1 1 2024-09-18T16:08:44.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-18T16:08:44.858 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'test' created 2024-09-18T16:08:44.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T16:08:45.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:191: TEST_scrub_snaps: wait_for_clean 2024-09-18T16:08:45.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:08:45.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:08:45.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:08:45.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:08:45.874 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:08:45.874 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:08:45.874 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:08:45.875 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:08:45.875 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:08:45.875 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:08:46.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:08:46.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:08:46.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:08:46.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:08:46.090 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:08:46.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T16:08:46.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:08:46.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:08:46.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:08:46.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836486 2024-09-18T16:08:46.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836486 2024-09-18T16:08:46.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836486' 2024-09-18T16:08:46.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:08:46.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836486 2024-09-18T16:08:46.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:08:46.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:08:46.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836486 2024-09-18T16:08:46.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:08:46.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836486 2024-09-18T16:08:46.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-09-18T16:08:46.634 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836486 2024-09-18T16:08:46.634 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:08:46.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836485 -lt 21474836486 2024-09-18T16:08:46.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:08:47.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:08:47.944 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:08:48.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836486 -lt 21474836486 2024-09-18T16:08:48.252 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:08:48.252 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:08:48.252 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:08:48.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:08:48.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:08:48.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:08:48.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:08:48.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:08:48.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:08:48.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:08:48.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:08:48.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:08:48.934 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:08:48.934 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:08:48.934 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:08:49.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:08:49.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:08:49.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:08:49.287 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: ceph osd dump 2024-09-18T16:08:49.288 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: grep '^pool.*['\'']test['\'']' 2024-09-18T16:08:49.288 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: awk '{ print $2 }' 2024-09-18T16:08:49.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: poolid=1 2024-09-18T16:08:49.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:194: TEST_scrub_snaps: dd if=/dev/urandom of=testdata.495962 bs=1032 count=1 2024-09-18T16:08:49.577 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records in 2024-09-18T16:08:49.577 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records out 2024-09-18T16:08:49.577 INFO:tasks.workunit.client.0.smithi046.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 0.000177322 s, 5.8 MB/s 2024-09-18T16:08:49.578 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: seq 1 16 2024-09-18T16:08:49.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-18T16:08:49.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj1 testdata.495962 2024-09-18T16:08:49.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-18T16:08:49.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj2 testdata.495962 2024-09-18T16:08:49.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-18T16:08:49.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj3 testdata.495962 2024-09-18T16:08:49.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-18T16:08:49.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj4 testdata.495962 2024-09-18T16:08:49.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-18T16:08:49.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj5 testdata.495962 2024-09-18T16:08:50.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-18T16:08:50.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj6 testdata.495962 2024-09-18T16:08:50.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-18T16:08:50.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj7 testdata.495962 2024-09-18T16:08:50.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-18T16:08:50.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj8 testdata.495962 2024-09-18T16:08:50.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-18T16:08:50.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj9 testdata.495962 2024-09-18T16:08:50.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-18T16:08:50.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj10 testdata.495962 2024-09-18T16:08:50.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-18T16:08:50.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj11 testdata.495962 2024-09-18T16:08:50.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-18T16:08:50.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj12 testdata.495962 2024-09-18T16:08:50.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-18T16:08:50.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj13 testdata.495962 2024-09-18T16:08:50.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-18T16:08:50.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj14 testdata.495962 2024-09-18T16:08:51.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-18T16:08:51.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj15 testdata.495962 2024-09-18T16:08:51.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-18T16:08:51.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj16 testdata.495962 2024-09-18T16:08:51.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:200: TEST_scrub_snaps: get_primary test obj1 2024-09-18T16:08:51.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T16:08:51.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T16:08:51.252 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T16:08:51.252 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T16:08:51.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:200: TEST_scrub_snaps: local primary=0 2024-09-18T16:08:51.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:202: TEST_scrub_snaps: create_scenario td/osd-scrub-snaps test testdata.495962 0 2024-09-18T16:08:51.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2024-09-18T16:08:51.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2024-09-18T16:08:51.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.495962 2024-09-18T16:08:51.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=0 2024-09-18T16:08:51.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2024-09-18T16:08:51.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2024-09-18T16:08:51.758 INFO:tasks.workunit.client.0.smithi046.stdout:created pool test snap snap1 2024-09-18T16:08:51.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.495962 bs=256 count=1 2024-09-18T16:08:51.764 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records in 2024-09-18T16:08:51.764 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records out 2024-09-18T16:08:51.764 INFO:tasks.workunit.client.0.smithi046.stderr:256 bytes copied, 0.000250868 s, 1.0 MB/s 2024-09-18T16:08:51.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.495962 2024-09-18T16:08:51.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.495962 2024-09-18T16:08:52.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.495962 2024-09-18T16:08:52.136 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2024-09-18T16:08:52.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:08:52.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.495962 2024-09-18T16:08:52.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:08:52.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.495962 2024-09-18T16:08:52.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:08:52.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.495962 2024-09-18T16:08:52.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:08:52.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.495962 2024-09-18T16:08:52.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:08:52.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.495962 2024-09-18T16:08:52.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:08:52.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.495962 2024-09-18T16:08:52.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:08:52.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.495962 2024-09-18T16:08:52.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:08:52.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.495962 2024-09-18T16:08:52.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:08:52.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.495962 2024-09-18T16:08:53.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2024-09-18T16:08:53.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2024-09-18T16:08:53.197 INFO:tasks.workunit.client.0.smithi046.stdout:created pool test snap snap2 2024-09-18T16:08:53.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.495962 bs=256 count=2 2024-09-18T16:08:53.203 INFO:tasks.workunit.client.0.smithi046.stderr:2+0 records in 2024-09-18T16:08:53.203 INFO:tasks.workunit.client.0.smithi046.stderr:2+0 records out 2024-09-18T16:08:53.203 INFO:tasks.workunit.client.0.smithi046.stderr:512 bytes copied, 0.000249688 s, 2.1 MB/s 2024-09-18T16:08:53.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.495962 2024-09-18T16:08:53.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2024-09-18T16:08:53.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2024-09-18T16:08:53.582 INFO:tasks.workunit.client.0.smithi046.stdout:created pool test snap snap3 2024-09-18T16:08:53.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.495962 bs=256 count=3 2024-09-18T16:08:53.587 INFO:tasks.workunit.client.0.smithi046.stderr:3+0 records in 2024-09-18T16:08:53.587 INFO:tasks.workunit.client.0.smithi046.stderr:3+0 records out 2024-09-18T16:08:53.588 INFO:tasks.workunit.client.0.smithi046.stderr:768 bytes copied, 0.000264461 s, 2.9 MB/s 2024-09-18T16:08:53.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.495962 2024-09-18T16:08:53.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2024-09-18T16:08:53.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2024-09-18T16:08:53.917 INFO:tasks.workunit.client.0.smithi046.stdout:created pool test snap snap4 2024-09-18T16:08:53.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.495962 bs=256 count=4 2024-09-18T16:08:53.923 INFO:tasks.workunit.client.0.smithi046.stderr:4+0 records in 2024-09-18T16:08:53.923 INFO:tasks.workunit.client.0.smithi046.stderr:4+0 records out 2024-09-18T16:08:53.923 INFO:tasks.workunit.client.0.smithi046.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000266888 s, 3.8 MB/s 2024-09-18T16:08:53.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.495962 2024-09-18T16:08:54.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.495962 2024-09-18T16:08:54.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2024-09-18T16:08:54.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2024-09-18T16:08:54.369 INFO:tasks.workunit.client.0.smithi046.stdout:created pool test snap snap5 2024-09-18T16:08:54.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2024-09-18T16:08:54.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2024-09-18T16:08:54.620 INFO:tasks.workunit.client.0.smithi046.stdout:created pool test snap snap6 2024-09-18T16:08:54.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.495962 bs=256 count=6 2024-09-18T16:08:54.625 INFO:tasks.workunit.client.0.smithi046.stderr:6+0 records in 2024-09-18T16:08:54.625 INFO:tasks.workunit.client.0.smithi046.stderr:6+0 records out 2024-09-18T16:08:54.625 INFO:tasks.workunit.client.0.smithi046.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 0.000279441 s, 5.5 MB/s 2024-09-18T16:08:54.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.495962 2024-09-18T16:08:54.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2024-09-18T16:08:54.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2024-09-18T16:08:54.963 INFO:tasks.workunit.client.0.smithi046.stdout:created pool test snap snap7 2024-09-18T16:08:54.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2024-09-18T16:08:55.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2024-09-18T16:08:55.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2024-09-18T16:08:55.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2024-09-18T16:08:55.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:08:55.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:08:55.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:08:55.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:08:55.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:08:55.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:08:55.595 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj1 2024-09-18T16:08:57.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:08:57.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2024-09-18T16:08:58.373 INFO:tasks.workunit.client.0.smithi046.stdout:WARNING: only removing head with clones present 2024-09-18T16:08:58.373 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:ff7b1f36:::obj1:head# 2024-09-18T16:08:58.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-09-18T16:08:58.934 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2024-09-18T16:09:00.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:09:00.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-18T16:09:00.901 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:c52c9666:::obj5:2# 2024-09-18T16:09:01.437 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-09-18T16:09:01.437 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2024-09-18T16:09:03.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:09:03.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:09:03.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-18T16:09:04.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-18T16:09:04.325 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:09:04.325 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-18T16:09:04.325 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:04.325 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-18T16:09:04.325 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-18T16:09:04.325 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-18T16:09:04.325 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-18T16:09:04.326 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-18T16:09:04.326 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-18T16:09:04.326 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-18T16:09:04.326 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:09:04.326 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-18T16:09:04.326 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-18T16:09:04.326 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-18T16:09:04.326 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-18T16:09:04.326 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:09:04.326 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-18T16:09:04.326 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:04.326 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:09:04.326 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:09:04.327 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-18T16:09:04.327 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:04.327 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-18T16:09:04.327 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:09:04.327 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:09:04.327 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:04.327 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-18T16:09:04.327 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:09:04.327 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:09:04.327 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:04.327 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-18T16:09:04.327 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:09:04.327 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:09:04.327 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:04.328 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-18T16:09:04.328 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:09:04.328 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:09:04.328 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:04.328 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:09:04.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-18T16:09:04.328 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-18T16:09:04.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-18T16:09:05.101 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:c52c9666:::obj5:1# 2024-09-18T16:09:05.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-18T16:09:06.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-18T16:09:06.949 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:09:06.949 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-18T16:09:06.949 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:06.949 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-18T16:09:06.949 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-18T16:09:06.949 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-18T16:09:06.949 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-18T16:09:06.949 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-18T16:09:06.950 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-18T16:09:06.950 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-18T16:09:06.950 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:09:06.950 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-18T16:09:06.950 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-18T16:09:06.950 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-18T16:09:06.950 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-18T16:09:06.950 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:09:06.950 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-18T16:09:06.950 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:06.950 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:09:06.950 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:09:06.950 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-18T16:09:06.951 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:06.951 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-18T16:09:06.951 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:09:06.951 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:09:06.951 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:06.951 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-18T16:09:06.951 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:09:06.951 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:09:06.951 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:06.951 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-18T16:09:06.951 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:09:06.951 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:09:06.951 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:06.952 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-18T16:09:06.952 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:09:06.952 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:09:06.952 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:06.952 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:09:06.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-18T16:09:06.952 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-18T16:09:06.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-09-18T16:09:06.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-09-18T16:09:06.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2024-09-18T16:09:08.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:09:08.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.495962 bs=256 count=18 2024-09-18T16:09:08.342 INFO:tasks.workunit.client.0.smithi046.stderr:18+0 records in 2024-09-18T16:09:08.342 INFO:tasks.workunit.client.0.smithi046.stderr:18+0 records out 2024-09-18T16:09:08.342 INFO:tasks.workunit.client.0.smithi046.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000410258 s, 11.2 MB/s 2024-09-18T16:09:08.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.495962 2024-09-18T16:09:09.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj3 2024-09-18T16:09:11.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:09:11.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.495962 bs=256 count=15 2024-09-18T16:09:11.194 INFO:tasks.workunit.client.0.smithi046.stderr:15+0 records in 2024-09-18T16:09:11.194 INFO:tasks.workunit.client.0.smithi046.stderr:15+0 records out 2024-09-18T16:09:11.194 INFO:tasks.workunit.client.0.smithi046.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000351695 s, 10.9 MB/s 2024-09-18T16:09:11.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.495962 2024-09-18T16:09:12.557 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj4 2024-09-18T16:09:12.557 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2024-09-18T16:09:13.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:09:13.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-18T16:09:14.718 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:0ee9ae15:::obj4:7# 2024-09-18T16:09:15.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-18T16:09:16.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-18T16:09:16.565 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-18T16:09:16.565 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:16.565 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-18T16:09:16.565 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-18T16:09:16.565 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-18T16:09:16.565 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-18T16:09:16.565 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-18T16:09:16.565 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-18T16:09:16.565 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-18T16:09:16.565 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:09:16.565 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-18T16:09:16.566 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-18T16:09:16.566 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-18T16:09:16.566 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-18T16:09:16.566 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-18T16:09:16.566 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:16.566 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:09:16.566 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-18T16:09:16.566 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:16.566 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-18T16:09:16.566 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:09:16.566 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:16.566 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-18T16:09:16.566 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:09:16.566 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:16.567 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-18T16:09:16.567 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:09:16.567 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:16.567 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-18T16:09:16.567 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:09:16.567 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:16.567 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:09:16.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-18T16:09:16.567 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:16.568 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:16.568 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:16.568 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:16.568 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:16.568 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:16.568 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:09:16.568 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj16 2024-09-18T16:09:16.569 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2024-09-18T16:09:17.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:09:17.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-18T16:09:18.474 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:461f8b5e:::obj16:7# 2024-09-18T16:09:19.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-18T16:09:20.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-18T16:09:20.577 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-18T16:09:20.577 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-18T16:09:20.577 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-18T16:09:20.577 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-18T16:09:20.577 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-18T16:09:20.577 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-18T16:09:20.577 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-18T16:09:20.577 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-18T16:09:20.577 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:09:20.578 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-18T16:09:20.578 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-18T16:09:20.578 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-18T16:09:20.578 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-18T16:09:20.578 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-18T16:09:20.578 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:09:20.578 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-18T16:09:20.578 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-18T16:09:20.578 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:09:20.578 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-18T16:09:20.578 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:09:20.578 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-18T16:09:20.578 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:09:20.579 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-18T16:09:20.579 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:09:20.579 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:09:20.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-18T16:09:20.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-09-18T16:09:20.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj2 2024-09-18T16:09:21.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:09:21.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2024-09-18T16:09:23.269 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:09:23.269 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2024-09-18T16:09:23.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:09:23.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.495962 bs=256 count=7 2024-09-18T16:09:23.274 INFO:tasks.workunit.client.0.smithi046.stderr:7+0 records in 2024-09-18T16:09:23.274 INFO:tasks.workunit.client.0.smithi046.stderr:7+0 records out 2024-09-18T16:09:23.274 INFO:tasks.workunit.client.0.smithi046.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 0.000296451 s, 6.0 MB/s 2024-09-18T16:09:23.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.495962 2024-09-18T16:09:25.108 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj6 2024-09-18T16:09:26.038 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:09:26.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:09:26.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2024-09-18T16:09:27.904 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj7 2024-09-18T16:09:28.780 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:09:29.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:09:29.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2024-09-18T16:09:30.608 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj8 2024-09-18T16:09:31.513 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:09:32.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:09:32.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2024-09-18T16:09:33.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj9 2024-09-18T16:09:34.331 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:09:34.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:09:34.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2024-09-18T16:09:36.276 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj10 2024-09-18T16:09:37.125 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:09:37.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:09:37.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2024-09-18T16:09:39.080 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj11 2024-09-18T16:09:39.964 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:09:40.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:09:40.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2024-09-18T16:09:41.824 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj12 2024-09-18T16:09:42.767 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:09:43.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:09:43.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2024-09-18T16:09:44.428 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj13 2024-09-18T16:09:45.330 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:09:46.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:09:46.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2024-09-18T16:09:47.429 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj14 2024-09-18T16:09:48.410 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:09:48.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:09:48.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2024-09-18T16:09:50.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2024-09-18T16:09:50.332 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj15 2024-09-18T16:09:51.198 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:09:51.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:09:51.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2024-09-18T16:09:52.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2024-09-18T16:09:52.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2024-09-18T16:09:52.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:204: TEST_scrub_snaps: rm -f testdata.495962 2024-09-18T16:09:52.880 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: expr 1 - 1 2024-09-18T16:09:52.882 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: seq 0 0 2024-09-18T16:09:52.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:09:52.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:208: TEST_scrub_snaps: activate_osd td/osd-scrub-snaps 0 2024-09-18T16:09:52.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-snaps 2024-09-18T16:09:52.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T16:09:52.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T16:09:52.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T16:09:52.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-snaps/0 2024-09-18T16:09:52.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-09-18T16:09:52.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:09:52.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:09:52.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:09:52.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-09-18T16:09:52.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-09-18T16:09:52.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T16:09:52.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T16:09:52.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-18T16:09:52.886 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T16:09:52.886 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:09:52.887 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:09:52.888 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:09:52.888 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:09:52.888 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:09:52.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.495962/$cluster-$name.asok' 2024-09-18T16:09:52.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' 2024-09-18T16:09:52.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:09:52.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-18T16:09:52.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-18T16:09:52.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:09:52.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:09:52.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:09:52.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:09:52.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T16:09:52.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T16:09:52.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-snaps/0 2024-09-18T16:09:52.890 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T16:09:52.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T16:09:52.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:09:52.891 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-snaps/0/whoami 2024-09-18T16:09:52.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T16:09:52.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T16:09:52.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T16:09:52.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T16:09:52.918 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:09:52.913+0000 7ff2e3cf3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:09:52.921 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:09:52.913+0000 7ff2e3cf3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:09:52.923 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:09:52.917+0000 7ff2e3cf3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:09:53.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T16:09:53.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:09:53.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:09:53.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:09:53.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:09:53.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:09:53.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:09:53.196 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:09:53.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:09:53.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:09:53.465 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:09:53.457+0000 7ff2e3cf3540 -1 Falling back to public interface 2024-09-18T16:09:53.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:09:54.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:09:54.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:09:54.548 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:09:54.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:09:54.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:09:54.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:09:54.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:09:55.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:09:55.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:09:55.861 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:09:55.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:09:55.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:09:55.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:09:56.094 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:09:56.089+0000 7ff2e3cf3540 -1 osd.0 18 log_to_monitors true 2024-09-18T16:09:56.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:09:57.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:09:57.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:09:57.181 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:09:57.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:09:57.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:09:57.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:09:57.489 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 21 up_thru 21 down_at 19 last_clean_interval [5,18) [v2:127.0.0.1:6802/515044,v1:127.0.0.1:6803/515044] [v2:127.0.0.1:6804/515044,v1:127.0.0.1:6805/515044] exists,up d53e95df-af76-4ea0-a7bf-cb6c49f50b24 2024-09-18T16:09:57.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:09:57.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:09:57.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:09:57.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:211: TEST_scrub_snaps: wait_for_clean 2024-09-18T16:09:57.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:09:57.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:09:57.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:09:57.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:09:57.490 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:09:57.491 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:09:57.491 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:09:57.491 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:09:57.492 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:09:57.492 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:09:57.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:09:57.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:09:57.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:09:57.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:09:57.703 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:09:58.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T16:09:58.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:09:58.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:09:58.009 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:09:58.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=90194313218 2024-09-18T16:09:58.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 90194313218 2024-09-18T16:09:58.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-90194313218' 2024-09-18T16:09:58.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:09:58.187 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-90194313218 2024-09-18T16:09:58.187 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:09:58.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:09:58.190 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-90194313218 2024-09-18T16:09:58.190 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:09:58.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=90194313218 2024-09-18T16:09:58.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 90194313218' 2024-09-18T16:09:58.192 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 90194313218 2024-09-18T16:09:58.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:09:58.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 90194313218 2024-09-18T16:09:58.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:09:59.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:09:59.502 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:09:59.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 90194313218 -lt 90194313218 2024-09-18T16:09:59.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:09:59.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:09:59.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:10:00.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:10:00.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:10:00.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:10:00.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:10:00.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:10:00.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:10:00.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:10:00.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:10:00.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:10:00.541 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:10:00.541 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:10:00.542 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:10:00.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:10:00.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:10:00.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:10:00.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:213: TEST_scrub_snaps: local pgid=1.0 2024-09-18T16:10:00.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:214: TEST_scrub_snaps: pg_scrub 1.0 2024-09-18T16:10:00.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: pg_scrub: local pgid=1.0 2024-09-18T16:10:00.906 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: pg_scrub: get_last_scrub_stamp 1.0 2024-09-18T16:10:00.907 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:10:00.907 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:10:00.907 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:10:00.907 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:10:01.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: pg_scrub: local last_scrub=2024-09-18T16:08:44.703668+0000 2024-09-18T16:10:01.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_scrub: ceph pg scrub 1.0 2024-09-18T16:10:01.547 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 1.0 on osd.0 to scrub 2024-09-18T16:10:01.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_scrub: wait_for_scrub 1.0 2024-09-18T16:08:44.703668+0000 2024-09-18T16:10:01.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-18T16:10:01.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T16:08:44.703668+0000 2024-09-18T16:10:01.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T16:10:01.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T16:10:01.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:10:01.557 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:10:01.557 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:10:01.557 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:10:01.557 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:10:01.557 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:10:01.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:08:44.703668+0000 '>' 2024-09-18T16:08:44.703668+0000 2024-09-18T16:10:01.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:10:02.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:10:02.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:10:02.854 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:10:02.854 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:10:02.855 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:10:02.855 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:10:02.855 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:10:03.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:08:44.703668+0000 '>' 2024-09-18T16:08:44.703668+0000 2024-09-18T16:10:03.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:10:04.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:10:04.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:10:04.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:10:04.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:10:04.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:10:04.162 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:10:04.162 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:10:04.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:08:44.703668+0000 '>' 2024-09-18T16:08:44.703668+0000 2024-09-18T16:10:04.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:10:05.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:10:05.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:10:05.470 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:10:05.470 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:10:05.470 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:10:05.471 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:10:05.471 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:10:05.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:08:44.703668+0000 '>' 2024-09-18T16:08:44.703668+0000 2024-09-18T16:10:05.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:10:06.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:10:06.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:10:06.781 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:10:06.781 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:10:06.781 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:10:06.781 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:10:06.781 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:10:07.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:08:44.703668+0000 '>' 2024-09-18T16:08:44.703668+0000 2024-09-18T16:10:07.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:10:08.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:10:08.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:10:08.091 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:10:08.091 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:10:08.091 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:10:08.092 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:10:08.092 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:10:08.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:10:02.128137+0000 '>' 2024-09-18T16:08:44.703668+0000 2024-09-18T16:10:08.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T16:10:08.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:218: TEST_scrub_snaps: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:10:08.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:218: TEST_scrub_snaps: wc -l 2024-09-18T16:10:08.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:218: TEST_scrub_snaps: test 2 = 2 2024-09-18T16:10:08.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:220: TEST_scrub_snaps: rados list-inconsistent-pg test 2024-09-18T16:10:08.488 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:222: TEST_scrub_snaps: jq '. | length' td/osd-scrub-snaps/json 2024-09-18T16:10:08.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:222: TEST_scrub_snaps: test 1 = 1 2024-09-18T16:10:08.530 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:224: TEST_scrub_snaps: jq -r '.[0]' td/osd-scrub-snaps/json 2024-09-18T16:10:08.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:224: TEST_scrub_snaps: test 1.0 = 1.0 2024-09-18T16:10:08.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:226: TEST_scrub_snaps: rados list-inconsistent-obj 1.0 2024-09-18T16:10:08.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:231: TEST_scrub_snaps: jq .inconsistents 2024-09-18T16:10:08.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:231: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-18T16:10:08.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:238: TEST_scrub_snaps: jq .inconsistents td/osd-scrub-snaps/json 2024-09-18T16:10:08.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:238: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-18T16:10:08.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:239: TEST_scrub_snaps: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-18T16:10:08.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-18T16:10:08.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:241: TEST_scrub_snaps: rados list-inconsistent-snapset 1.0 2024-09-18T16:10:08.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:243: TEST_scrub_snaps: jq .inconsistents 2024-09-18T16:10:08.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:243: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-18T16:10:08.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:622: TEST_scrub_snaps: jq .inconsistents td/osd-scrub-snaps/json 2024-09-18T16:10:08.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:622: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-18T16:10:08.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:623: TEST_scrub_snaps: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-18T16:10:08.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-18T16:10:08.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:624: TEST_scrub_snaps: test no = yes 2024-09-18T16:10:08.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:629: TEST_scrub_snaps: test '' = yes 2024-09-18T16:10:08.824 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:634: TEST_scrub_snaps: find td/osd-scrub-snaps 2024-09-18T16:10:08.824 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:634: TEST_scrub_snaps: grep 'osd[^/]*\.pid' 2024-09-18T16:10:08.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:634: TEST_scrub_snaps: pidfiles=td/osd-scrub-snaps/osd.0.pid 2024-09-18T16:10:08.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:635: TEST_scrub_snaps: pids= 2024-09-18T16:10:08.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:636: TEST_scrub_snaps: for pidfile in ${pidfiles} 2024-09-18T16:10:08.827 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:638: TEST_scrub_snaps: cat td/osd-scrub-snaps/osd.0.pid 2024-09-18T16:10:08.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:638: TEST_scrub_snaps: pids+='515044 ' 2024-09-18T16:10:08.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:641: TEST_scrub_snaps: ERRORS=0 2024-09-18T16:10:08.828 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:643: TEST_scrub_snaps: seq 1 7 2024-09-18T16:10:08.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:643: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-18T16:10:08.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:645: TEST_scrub_snaps: rados -p test rmsnap snap1 2024-09-18T16:10:08.994 INFO:tasks.workunit.client.0.smithi046.stdout:removed pool test snap snap1 2024-09-18T16:10:08.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:643: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-18T16:10:08.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:645: TEST_scrub_snaps: rados -p test rmsnap snap2 2024-09-18T16:10:09.233 INFO:tasks.workunit.client.0.smithi046.stdout:removed pool test snap snap2 2024-09-18T16:10:09.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:643: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-18T16:10:09.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:645: TEST_scrub_snaps: rados -p test rmsnap snap3 2024-09-18T16:10:09.468 INFO:tasks.workunit.client.0.smithi046.stdout:removed pool test snap snap3 2024-09-18T16:10:09.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:643: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-18T16:10:09.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:645: TEST_scrub_snaps: rados -p test rmsnap snap4 2024-09-18T16:10:09.727 INFO:tasks.workunit.client.0.smithi046.stdout:removed pool test snap snap4 2024-09-18T16:10:09.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:643: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-18T16:10:09.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:645: TEST_scrub_snaps: rados -p test rmsnap snap5 2024-09-18T16:10:09.944 INFO:tasks.workunit.client.0.smithi046.stdout:removed pool test snap snap5 2024-09-18T16:10:09.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:643: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-18T16:10:09.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:645: TEST_scrub_snaps: rados -p test rmsnap snap6 2024-09-18T16:10:10.212 INFO:tasks.workunit.client.0.smithi046.stdout:removed pool test snap snap6 2024-09-18T16:10:10.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:643: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-18T16:10:10.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:645: TEST_scrub_snaps: rados -p test rmsnap snap7 2024-09-18T16:10:10.429 INFO:tasks.workunit.client.0.smithi046.stdout:removed pool test snap snap7 2024-09-18T16:10:10.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:647: TEST_scrub_snaps: sleep 5 2024-09-18T16:10:15.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:648: TEST_scrub_snaps: local -i loop=0 2024-09-18T16:10:15.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:649: TEST_scrub_snaps: ceph pg dump pgs 2024-09-18T16:10:15.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:649: TEST_scrub_snaps: grep -q snaptrim 2024-09-18T16:10:15.731 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:10:15.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:651: TEST_scrub_snaps: ceph pg dump pgs 2024-09-18T16:10:15.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:651: TEST_scrub_snaps: grep -q snaptrim_error 2024-09-18T16:10:16.029 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:10:16.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:653: TEST_scrub_snaps: break 2024-09-18T16:10:16.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:662: TEST_scrub_snaps: ceph pg dump pgs 2024-09-18T16:10:16.346 INFO:tasks.workunit.client.0.smithi046.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T16:10:16.347 INFO:tasks.workunit.client.0.smithi046.stdout:1.0 35 0 0 0 0 23416 0 0 60 0 60 active+clean+inconsistent+snaptrim_error 2024-09-18T16:10:10.453167+0000 23'60 29:108 [0] 0 [0] 0 18'56 2024-09-18T16:10:02.128137+0000 0'0 2024-09-18T16:08:44.703668+0000 7 1 periodic scrub scheduled @ 2024-09-19T23:56:06.264442+0000 0 0 2024-09-18T16:10:16.347 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-09-18T16:10:16.347 INFO:tasks.workunit.client.0.smithi046.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T16:10:16.350 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:10:16.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:664: TEST_scrub_snaps: for pid in $pids 2024-09-18T16:10:16.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:666: TEST_scrub_snaps: kill -0 515044 2024-09-18T16:10:16.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:673: TEST_scrub_snaps: kill_daemons td/osd-scrub-snaps 2024-09-18T16:10:16.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:10:16.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:10:16.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:10:16.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:10:16.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:10:21.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:10:21.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:675: TEST_scrub_snaps: declare -a err_strings 2024-09-18T16:10:21.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:676: TEST_scrub_snaps: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj10:.* : is missing in clone_overlap' 2024-09-18T16:10:21.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:677: TEST_scrub_snaps: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : no '\''_'\'' attr' 2024-09-18T16:10:21.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:678: TEST_scrub_snaps: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : is an unexpected clone' 2024-09-18T16:10:21.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:679: TEST_scrub_snaps: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:4 : on disk size [(]4608[)] does not match object info size [(]512[)] adjusted for ondisk to [(]512[)]' 2024-09-18T16:10:21.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:680: TEST_scrub_snaps: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:2' 2024-09-18T16:10:21.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:681: TEST_scrub_snaps: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:1' 2024-09-18T16:10:21.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:682: TEST_scrub_snaps: err_strings[6]='log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj5:head : 2 missing clone[(]s[)]' 2024-09-18T16:10:21.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:683: TEST_scrub_snaps: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj8:head : snaps.seq not set' 2024-09-18T16:10:21.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:684: TEST_scrub_snaps: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj7:1 : is an unexpected clone' 2024-09-18T16:10:21.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:685: TEST_scrub_snaps: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj3:head : on disk size [(]3840[)] does not match object info size [(]768[)] adjusted for ondisk to [(]768[)]' 2024-09-18T16:10:21.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:686: TEST_scrub_snaps: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj6:1 : is an unexpected clone' 2024-09-18T16:10:21.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:687: TEST_scrub_snaps: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:head : no '\''snapset'\'' attr' 2024-09-18T16:10:21.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:688: TEST_scrub_snaps: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:7 : clone ignored due to missing snapset' 2024-09-18T16:10:21.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:689: TEST_scrub_snaps: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:4 : clone ignored due to missing snapset' 2024-09-18T16:10:21.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:690: TEST_scrub_snaps: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj4:head : expected clone .*:::obj4:7' 2024-09-18T16:10:21.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:691: TEST_scrub_snaps: err_strings[15]='log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj4:head : 1 missing clone[(]s[)]' 2024-09-18T16:10:21.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:692: TEST_scrub_snaps: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj1:1 : is an unexpected clone' 2024-09-18T16:10:21.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:693: TEST_scrub_snaps: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj9:1 : is missing in clone_size' 2024-09-18T16:10:21.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:694: TEST_scrub_snaps: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj11:1 : is an unexpected clone' 2024-09-18T16:10:21.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:695: TEST_scrub_snaps: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj14:1 : size 1032 != clone_size 1033' 2024-09-18T16:10:21.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:696: TEST_scrub_snaps: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' 2024-09-18T16:10:21.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:697: TEST_scrub_snaps: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj15:head : can'\''t decode '\''snapset'\'' attr ' 2024-09-18T16:10:21.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:698: TEST_scrub_snaps: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2024-09-18T16:10:21.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-18T16:10:21.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj10:.* : is missing in clone_overlap' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:10:21.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-18T16:10:21.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : no '\''_'\'' attr' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:10:21.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-18T16:10:21.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:10:21.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-18T16:10:21.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:4 : on disk size [(]4608[)] does not match object info size [(]512[)] adjusted for ondisk to [(]512[)]' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:10:21.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-18T16:10:21.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:2' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:10:21.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-18T16:10:21.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:1' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:10:21.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-18T16:10:21.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj5:head : 2 missing clone[(]s[)]' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:10:21.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-18T16:10:21.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj8:head : snaps.seq not set' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:10:21.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-18T16:10:21.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj7:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:10:21.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-18T16:10:21.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj3:head : on disk size [(]3840[)] does not match object info size [(]768[)] adjusted for ondisk to [(]768[)]' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:10:21.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-18T16:10:21.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj6:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:10:21.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-18T16:10:21.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:head : no '\''snapset'\'' attr' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:10:21.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-18T16:10:21.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:7 : clone ignored due to missing snapset' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:10:21.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-18T16:10:21.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:4 : clone ignored due to missing snapset' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:10:21.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-18T16:10:21.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj4:head : expected clone .*:::obj4:7' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:10:21.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-18T16:10:21.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj4:head : 1 missing clone[(]s[)]' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:10:21.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-18T16:10:21.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj1:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:10:21.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-18T16:10:21.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj9:1 : is missing in clone_size' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:10:21.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-18T16:10:21.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj11:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:10:21.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-18T16:10:21.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj14:1 : size 1032 != clone_size 1033' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:10:21.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-18T16:10:21.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:10:21.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-18T16:10:21.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj15:head : can'\''t decode '\''snapset'\'' attr ' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:10:21.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-18T16:10:21.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:10:21.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:709: TEST_scrub_snaps: '[' 0 '!=' 0 ']' 2024-09-18T16:10:21.780 INFO:tasks.workunit.client.0.smithi046.stdout:TEST PASSED 2024-09-18T16:10:21.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:715: TEST_scrub_snaps: echo 'TEST PASSED' 2024-09-18T16:10:21.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:716: TEST_scrub_snaps: return 0 2024-09-18T16:10:21.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2024-09-18T16:10:21.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-09-18T16:10:21.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:10:21.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-18T16:10:21.782 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:10:21.782 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:10:21.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:10:21.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:10:21.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:10:21.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:10:21.794 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:10:21.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:10:21.797 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:10:21.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:10:21.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:10:21.800 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:10:21.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:10:21.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:10:21.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:10:21.803 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:10:21.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:10:21.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:10:21.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-09-18T16:10:21.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:10:21.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:10:21.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:10:21.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.495962 2024-09-18T16:10:21.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:10:21.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:10:21.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2024-09-18T16:10:21.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2024-09-18T16:10:21.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-snaps 2024-09-18T16:10:21.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-snaps 2024-09-18T16:10:21.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-09-18T16:10:21.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:10:21.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-18T16:10:21.826 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:10:21.826 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:10:21.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:10:21.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:10:21.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:10:21.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:10:21.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:10:21.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:10:21.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:10:21.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:10:21.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:10:21.836 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:10:21.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:10:21.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:10:21.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:10:21.838 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:10:21.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:10:21.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:10:21.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-09-18T16:10:21.844 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:10:21.844 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:10:21.844 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:10:21.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.495962 2024-09-18T16:10:21.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:10:21.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:10:21.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-snaps 2024-09-18T16:10:21.849 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T16:10:21.849 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:10:21.849 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:10:21.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.495962 2024-09-18T16:10:21.851 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T16:10:21.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T16:10:21.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T16:10:21.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2024-09-18T16:10:21.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps_primary td/osd-scrub-snaps 2024-09-18T16:10:21.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1160: TEST_scrub_snaps_primary: local dir=td/osd-scrub-snaps 2024-09-18T16:10:21.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1161: TEST_scrub_snaps_primary: ORIG_ARGS='--fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-09-18T16:10:21.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1162: TEST_scrub_snaps_primary: CEPH_ARGS+=' --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3' 2024-09-18T16:10:21.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1163: TEST_scrub_snaps_primary: _scrub_snaps_multi td/osd-scrub-snaps primary 2024-09-18T16:10:21.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:720: _scrub_snaps_multi: local dir=td/osd-scrub-snaps 2024-09-18T16:10:21.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:721: _scrub_snaps_multi: local poolname=test 2024-09-18T16:10:21.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:722: _scrub_snaps_multi: local OBJS=16 2024-09-18T16:10:21.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:723: _scrub_snaps_multi: local OSDS=2 2024-09-18T16:10:21.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:724: _scrub_snaps_multi: local which=primary 2024-09-18T16:10:21.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:726: _scrub_snaps_multi: TESTDATA=testdata.495962 2024-09-18T16:10:21.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:728: _scrub_snaps_multi: run_mon td/osd-scrub-snaps a --osd_pool_default_size=2 2024-09-18T16:10:21.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-snaps 2024-09-18T16:10:21.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T16:10:21.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T16:10:21.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T16:10:21.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-snaps/a 2024-09-18T16:10:21.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=2 2024-09-18T16:10:22.243 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T16:10:22.243 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:10:22.243 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:10:22.243 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:10:22.244 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:10:22.244 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:10:22.244 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.495962/$cluster-$name.asok' 2024-09-18T16:10:22.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-18T16:10:22.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T16:10:22.517 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T16:10:22.517 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:10:22.517 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:10:22.517 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T16:10:22.518 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T16:10:22.518 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:10:22.518 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:10:22.518 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:10:22.519 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:10:22.519 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:10:22.519 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:10:22.520 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.495962/ceph-mon.a.asok 2024-09-18T16:10:22.520 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:10:22.520 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.495962/ceph-mon.a.asok config get fsid 2024-09-18T16:10:22.613 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T16:10:22.613 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:10:22.613 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:10:22.613 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T16:10:22.614 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T16:10:22.614 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:10:22.615 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:10:22.615 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:10:22.615 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:10:22.615 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:10:22.615 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:10:22.615 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.495962/ceph-mon.a.asok 2024-09-18T16:10:22.615 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:10:22.616 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.495962/ceph-mon.a.asok config get mon_host 2024-09-18T16:10:22.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:729: _scrub_snaps_multi: run_mgr td/osd-scrub-snaps x 2024-09-18T16:10:22.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-snaps 2024-09-18T16:10:22.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T16:10:22.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T16:10:22.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T16:10:22.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-snaps/x 2024-09-18T16:10:22.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T16:10:23.577 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T16:10:23.577 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:10:23.577 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:10:23.578 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:10:23.578 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:10:23.578 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:10:23.578 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.495962/$cluster-$name.asok' 2024-09-18T16:10:23.579 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:10:23.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:10:23.620 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:730: _scrub_snaps_multi: expr 2 - 1 2024-09-18T16:10:23.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:730: _scrub_snaps_multi: seq 0 1 2024-09-18T16:10:23.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:730: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:10:23.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:732: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 0 2024-09-18T16:10:23.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-snaps 2024-09-18T16:10:23.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:10:23.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T16:10:23.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:10:23.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-snaps/0 2024-09-18T16:10:23.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3' 2024-09-18T16:10:23.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:10:23.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:10:23.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:10:23.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-09-18T16:10:23.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-09-18T16:10:23.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:10:23.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:10:23.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-18T16:10:23.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:10:23.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:10:23.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:10:23.627 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:10:23.627 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:10:23.627 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:10:23.629 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.495962/$cluster-$name.asok' 2024-09-18T16:10:23.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' 2024-09-18T16:10:23.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:10:23.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:10:23.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:10:23.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-18T16:10:23.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-18T16:10:23.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:10:23.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:10:23.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:10:23.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:10:23.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:10:23.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:10:23.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-snaps/0 2024-09-18T16:10:23.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:10:23.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a991ad25-2dc7-4b32-8aca-6d9121ad4e4f 2024-09-18T16:10:23.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a991ad25-2dc7-4b32-8aca-6d9121ad4e4f' 2024-09-18T16:10:23.631 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 a991ad25-2dc7-4b32-8aca-6d9121ad4e4f 2024-09-18T16:10:23.631 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:10:23.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBv++pmbzG7JhAAAqbFV0Vedgef5ycjIInKuQ== 2024-09-18T16:10:23.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBv++pmbzG7JhAAAqbFV0Vedgef5ycjIInKuQ=="}' 2024-09-18T16:10:23.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a991ad25-2dc7-4b32-8aca-6d9121ad4e4f -i td/osd-scrub-snaps/0/new.json 2024-09-18T16:10:23.933 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:10:23.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-snaps/0/new.json 2024-09-18T16:10:23.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBv++pmbzG7JhAAAqbFV0Vedgef5ycjIInKuQ== --osd-uuid a991ad25-2dc7-4b32-8aca-6d9121ad4e4f 2024-09-18T16:10:23.982 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:10:23.977+0000 7f8a37ef2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:10:24.028 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:10:24.021+0000 7f8a37ef2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:10:24.029 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:10:24.021+0000 7f8a37ef2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:10:24.030 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:10:24.025+0000 7f8a37ef2540 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-09-18T16:10:24.030 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:10:24.025+0000 7f8a37ef2540 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-09-18T16:10:24.030 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:10:24.025+0000 7f8a37ef2540 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-09-18T16:10:24.074 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:10:24.069+0000 7f8a37ef2540 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2024-09-18T16:10:30.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2024-09-18T16:10:30.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:10:30.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T16:10:30.213 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T16:10:30.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:10:30.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T16:10:30.594 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T16:10:30.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:10:30.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:10:30.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:10:30.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:10:30.628 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:10:30.621+0000 7f18a5dcd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:10:30.631 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:10:30.625+0000 7f18a5dcd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:10:30.634 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:10:30.625+0000 7f18a5dcd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:10:30.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T16:10:30.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:10:30.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:10:30.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:10:30.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:10:30.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:30.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:10:30.891 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:10:30.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:30.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:31.195 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:10:31.189+0000 7f18a5dcd540 -1 Falling back to public interface 2024-09-18T16:10:31.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:32.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:32.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:32.223 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:10:32.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:10:32.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:32.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:32.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:33.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:33.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:33.538 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:10:33.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:10:33.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:33.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:33.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:33.975 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:10:33.969+0000 7f18a5dcd540 -1 osd.0 0 log_to_monitors true 2024-09-18T16:10:34.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:34.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:34.862 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:10:34.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:10:34.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:34.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:35.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:36.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:36.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:36.182 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:10:36.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:10:36.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:36.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:36.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:37.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:37.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:37.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:10:37.509 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:10:37.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:37.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:37.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:38.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:38.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:38.824 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:10:38.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:10:38.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:38.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:39.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:40.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:40.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:40.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:10:40.140 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:10:40.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:40.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:40.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:41.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:41.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:41.461 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:10:41.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:10:41.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:41.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:41.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:42.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:42.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:42.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:10:42.812 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:10:42.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:42.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:43.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:44.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:44.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:44.133 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:10:44.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:10:44.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:44.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:44.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:45.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:45.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:45.454 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:10:45.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:10:45.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:45.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:46.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:47.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:47.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:47.207 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:10:47.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:10:47.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:47.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:47.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:48.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:48.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:48.781 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:10:48.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:10:48.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:48.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:49.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:50.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:50.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:50.550 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:10:50.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:10:50.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:50.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:51.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:52.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:52.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:52.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:10:52.100 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:10:52.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:52.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:52.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:53.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:53.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:53.707 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:10:53.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:10:53.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:53.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:54.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:55.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:55.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:55.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:10:55.364 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:10:55.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:55.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:55.690 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/517560,v1:127.0.0.1:6803/517560] [v2:127.0.0.1:6804/517560,v1:127.0.0.1:6805/517560] exists,up a991ad25-2dc7-4b32-8aca-6d9121ad4e4f 2024-09-18T16:10:55.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:10:55.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:10:55.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:10:55.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:730: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:10:55.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:732: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 1 2024-09-18T16:10:55.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-snaps 2024-09-18T16:10:55.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:10:55.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T16:10:55.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:10:55.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-snaps/1 2024-09-18T16:10:55.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3' 2024-09-18T16:10:55.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:10:55.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:10:55.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:10:55.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-09-18T16:10:55.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-09-18T16:10:55.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:10:55.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:10:55.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-18T16:10:55.694 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:10:55.694 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:10:55.694 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:10:55.694 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:10:55.694 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:10:55.695 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:10:55.696 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.495962/$cluster-$name.asok' 2024-09-18T16:10:55.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' 2024-09-18T16:10:55.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:10:55.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:10:55.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:10:55.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-18T16:10:55.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-18T16:10:55.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:10:55.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:10:55.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:10:55.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:10:55.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:10:55.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:10:55.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-snaps/1 2024-09-18T16:10:55.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:10:55.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=45b5bd55-b062-42d9-a2f7-656472781481 2024-09-18T16:10:55.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 45b5bd55-b062-42d9-a2f7-656472781481' 2024-09-18T16:10:55.700 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 45b5bd55-b062-42d9-a2f7-656472781481 2024-09-18T16:10:55.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:10:55.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCP++pmEr/QKhAAl9dQm6aiTYdS9SXfOSQ/0w== 2024-09-18T16:10:55.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCP++pmEr/QKhAAl9dQm6aiTYdS9SXfOSQ/0w=="}' 2024-09-18T16:10:55.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 45b5bd55-b062-42d9-a2f7-656472781481 -i td/osd-scrub-snaps/1/new.json 2024-09-18T16:10:56.213 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:10:56.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-snaps/1/new.json 2024-09-18T16:10:56.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCP++pmEr/QKhAAl9dQm6aiTYdS9SXfOSQ/0w== --osd-uuid 45b5bd55-b062-42d9-a2f7-656472781481 2024-09-18T16:10:56.263 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:10:56.257+0000 7fcd07709540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:10:56.271 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:10:56.265+0000 7fcd07709540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:10:56.273 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:10:56.265+0000 7fcd07709540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:10:56.274 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:10:56.269+0000 7fcd07709540 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2024-09-18T16:10:56.274 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:10:56.269+0000 7fcd07709540 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2024-09-18T16:10:56.274 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:10:56.269+0000 7fcd07709540 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2024-09-18T16:10:56.347 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:10:56.341+0000 7fcd07709540 -1 bluestore(td/osd-scrub-snaps/1) _read_fsid unparsable uuid 2024-09-18T16:11:02.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-snaps/1/keyring 2024-09-18T16:11:02.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:11:02.625 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T16:11:02.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T16:11:02.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-snaps/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:11:03.013 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T16:11:03.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T16:11:03.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:11:03.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:11:03.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:11:03.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:11:03.047 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:11:03.041+0000 7fadf0417540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:11:03.050 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:11:03.045+0000 7fadf0417540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:11:03.051 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:11:03.045+0000 7fadf0417540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:11:03.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T16:11:03.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:11:03.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T16:11:03.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:11:03.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:11:03.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:03.358 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:11:03.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:11:03.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:03.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:03.603 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:11:03.597+0000 7fadf0417540 -1 Falling back to public interface 2024-09-18T16:11:03.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:04.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:04.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:04.720 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:11:04.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:11:04.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:04.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:05.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:06.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:06.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:06.034 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:11:06.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:11:06.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:06.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:06.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:06.433 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:11:06.425+0000 7fadf0417540 -1 osd.1 0 log_to_monitors true 2024-09-18T16:11:07.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:07.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:07.357 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:11:07.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:11:07.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:07.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:07.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:08.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:08.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:08.795 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:11:08.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:11:08.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:08.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:09.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:10.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:10.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:10.186 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:11:10.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:11:10.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:10.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:10.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:11.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:11.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:11.547 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:11:11.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:11:11.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:11.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:11.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:12.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:12.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:12.970 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:11:12.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:11:12.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:12.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:13.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:14.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:14.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:14.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:11:14.379 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:11:14.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:14.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:14.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:15.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:15.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:15.707 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:11:15.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:11:15.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:15.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:16.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:17.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:17.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:17.138 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:11:17.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:11:17.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:17.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:17.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:18.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:18.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:18.550 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:11:18.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:11:18.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:18.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:18.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:19.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:19.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:19.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:11:19.883 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:11:19.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:19.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:20.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:21.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:21.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:21.448 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:11:21.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:11:21.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:21.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:21.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:22.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:22.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:22.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:11:22.782 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:11:22.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:22.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:23.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:24.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:24.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:24.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:11:24.128 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:11:24.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:24.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:24.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:25.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:25.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:25.490 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:11:25.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:11:25.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:25.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:25.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:26.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:26.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:26.864 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:11:26.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:11:26.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:26.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:27.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:28.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:28.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:28.494 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:11:28.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:11:28.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:28.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:28.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:29.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:29.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:29.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T16:11:29.880 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T16:11:29.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:29.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:30.238 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/519314,v1:127.0.0.1:6811/519314] [v2:127.0.0.1:6812/519314,v1:127.0.0.1:6813/519314] exists,up 45b5bd55-b062-42d9-a2f7-656472781481 2024-09-18T16:11:30.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:11:30.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:11:30.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:11:30.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:736: _scrub_snaps_multi: ceph osd set noscrub 2024-09-18T16:11:30.731 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T16:11:30.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:737: _scrub_snaps_multi: ceph osd set nodeep-scrub 2024-09-18T16:11:31.317 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T16:11:31.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: create_pool test 1 1 2024-09-18T16:11:31.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-18T16:11:31.875 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'test' created 2024-09-18T16:11:31.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T16:11:32.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:741: _scrub_snaps_multi: wait_for_clean 2024-09-18T16:11:32.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:11:32.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:11:32.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:11:32.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:11:32.892 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:11:32.893 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:11:32.893 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:11:32.893 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:11:32.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:11:32.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:11:33.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:11:33.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:11:33.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:11:33.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:11:33.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:11:33.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:11:33.439 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T16:11:33.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:11:33.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:11:33.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:11:33.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836493 2024-09-18T16:11:33.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836493 2024-09-18T16:11:33.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836493' 2024-09-18T16:11:33.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:11:33.617 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:11:33.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672963 2024-09-18T16:11:33.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672963 2024-09-18T16:11:33.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836493 1-42949672963' 2024-09-18T16:11:33.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:11:33.766 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836493 2024-09-18T16:11:33.767 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:11:33.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:11:33.769 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836493 2024-09-18T16:11:33.769 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:11:33.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836493 2024-09-18T16:11:33.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-09-18T16:11:33.770 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836493 2024-09-18T16:11:33.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:11:34.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836492 -lt 21474836493 2024-09-18T16:11:34.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:11:35.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:11:35.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:11:35.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836493 -lt 21474836493 2024-09-18T16:11:35.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:11:35.402 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672963 2024-09-18T16:11:35.403 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:11:35.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:11:35.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672963 2024-09-18T16:11:35.406 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:11:35.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672963 2024-09-18T16:11:35.408 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672963 2024-09-18T16:11:35.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2024-09-18T16:11:35.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:11:35.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672963 -lt 42949672963 2024-09-18T16:11:35.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:11:35.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:11:35.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:11:36.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:11:36.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:11:36.147 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:11:36.147 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:11:36.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:11:36.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:11:36.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:11:36.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:11:36.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:11:36.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:11:36.496 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:11:36.496 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:11:36.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:11:36.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:11:36.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:11:36.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: ceph osd dump 2024-09-18T16:11:36.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: grep '^pool.*['\'']test['\'']' 2024-09-18T16:11:36.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: awk '{ print $2 }' 2024-09-18T16:11:37.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: poolid=1 2024-09-18T16:11:37.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: dd if=/dev/urandom of=testdata.495962 bs=1032 count=1 2024-09-18T16:11:37.249 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records in 2024-09-18T16:11:37.249 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records out 2024-09-18T16:11:37.249 INFO:tasks.workunit.client.0.smithi046.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 0.000195388 s, 5.3 MB/s 2024-09-18T16:11:37.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: seq 1 16 2024-09-18T16:11:37.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:11:37.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj1 testdata.495962 2024-09-18T16:11:37.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:11:37.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj2 testdata.495962 2024-09-18T16:11:37.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:11:37.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj3 testdata.495962 2024-09-18T16:11:37.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:11:37.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj4 testdata.495962 2024-09-18T16:11:37.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:11:37.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj5 testdata.495962 2024-09-18T16:11:38.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:11:38.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj6 testdata.495962 2024-09-18T16:11:38.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:11:38.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj7 testdata.495962 2024-09-18T16:11:38.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:11:38.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj8 testdata.495962 2024-09-18T16:11:38.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:11:38.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj9 testdata.495962 2024-09-18T16:11:38.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:11:38.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj10 testdata.495962 2024-09-18T16:11:38.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:11:38.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj11 testdata.495962 2024-09-18T16:11:38.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:11:38.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj12 testdata.495962 2024-09-18T16:11:39.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:11:39.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj13 testdata.495962 2024-09-18T16:11:39.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:11:39.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj14 testdata.495962 2024-09-18T16:11:39.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:11:39.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj15 testdata.495962 2024-09-18T16:11:39.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:11:39.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj16 testdata.495962 2024-09-18T16:11:39.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:750: _scrub_snaps_multi: get_primary test obj1 2024-09-18T16:11:39.629 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T16:11:39.629 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T16:11:39.629 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T16:11:39.629 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T16:11:39.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:750: _scrub_snaps_multi: local primary=1 2024-09-18T16:11:39.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:751: _scrub_snaps_multi: get_not_primary test obj1 2024-09-18T16:11:39.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-09-18T16:11:39.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-09-18T16:11:39.994 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-09-18T16:11:39.994 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T16:11:39.994 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T16:11:39.994 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T16:11:39.995 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T16:11:40.348 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T16:11:40.349 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-09-18T16:11:40.349 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T16:11:40.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:751: _scrub_snaps_multi: local replica=0 2024-09-18T16:11:40.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:753: _scrub_snaps_multi: eval create_scenario td/osd-scrub-snaps test testdata.495962 '$primary' 2024-09-18T16:11:40.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:753: _scrub_snaps_multi: create_scenario td/osd-scrub-snaps test testdata.495962 1 2024-09-18T16:11:40.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2024-09-18T16:11:40.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2024-09-18T16:11:40.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.495962 2024-09-18T16:11:40.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=1 2024-09-18T16:11:40.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2024-09-18T16:11:40.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2024-09-18T16:11:40.904 INFO:tasks.workunit.client.0.smithi046.stdout:created pool test snap snap1 2024-09-18T16:11:40.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.495962 bs=256 count=1 2024-09-18T16:11:40.910 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records in 2024-09-18T16:11:40.910 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records out 2024-09-18T16:11:40.910 INFO:tasks.workunit.client.0.smithi046.stderr:256 bytes copied, 0.000251911 s, 1.0 MB/s 2024-09-18T16:11:40.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.495962 2024-09-18T16:11:41.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.495962 2024-09-18T16:11:41.332 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.495962 2024-09-18T16:11:41.484 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2024-09-18T16:11:41.485 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:11:41.485 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.495962 2024-09-18T16:11:41.608 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:11:41.608 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.495962 2024-09-18T16:11:41.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:11:41.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.495962 2024-09-18T16:11:41.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:11:41.902 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.495962 2024-09-18T16:11:42.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:11:42.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.495962 2024-09-18T16:11:42.136 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:11:42.136 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.495962 2024-09-18T16:11:42.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:11:42.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.495962 2024-09-18T16:11:42.395 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:11:42.395 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.495962 2024-09-18T16:11:42.529 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:11:42.529 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.495962 2024-09-18T16:11:42.680 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2024-09-18T16:11:42.680 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2024-09-18T16:11:42.846 INFO:tasks.workunit.client.0.smithi046.stdout:created pool test snap snap2 2024-09-18T16:11:42.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.495962 bs=256 count=2 2024-09-18T16:11:42.852 INFO:tasks.workunit.client.0.smithi046.stderr:2+0 records in 2024-09-18T16:11:42.852 INFO:tasks.workunit.client.0.smithi046.stderr:2+0 records out 2024-09-18T16:11:42.852 INFO:tasks.workunit.client.0.smithi046.stderr:512 bytes copied, 0.000248201 s, 2.1 MB/s 2024-09-18T16:11:42.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.495962 2024-09-18T16:11:43.115 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2024-09-18T16:11:43.115 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2024-09-18T16:11:43.297 INFO:tasks.workunit.client.0.smithi046.stdout:created pool test snap snap3 2024-09-18T16:11:43.301 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.495962 bs=256 count=3 2024-09-18T16:11:43.303 INFO:tasks.workunit.client.0.smithi046.stderr:3+0 records in 2024-09-18T16:11:43.303 INFO:tasks.workunit.client.0.smithi046.stderr:3+0 records out 2024-09-18T16:11:43.303 INFO:tasks.workunit.client.0.smithi046.stderr:768 bytes copied, 0.000270235 s, 2.8 MB/s 2024-09-18T16:11:43.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.495962 2024-09-18T16:11:43.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2024-09-18T16:11:43.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2024-09-18T16:11:43.788 INFO:tasks.workunit.client.0.smithi046.stdout:created pool test snap snap4 2024-09-18T16:11:43.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.495962 bs=256 count=4 2024-09-18T16:11:43.794 INFO:tasks.workunit.client.0.smithi046.stderr:4+0 records in 2024-09-18T16:11:43.794 INFO:tasks.workunit.client.0.smithi046.stderr:4+0 records out 2024-09-18T16:11:43.794 INFO:tasks.workunit.client.0.smithi046.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000267331 s, 3.8 MB/s 2024-09-18T16:11:43.794 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.495962 2024-09-18T16:11:44.027 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.495962 2024-09-18T16:11:44.195 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2024-09-18T16:11:44.195 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2024-09-18T16:11:44.361 INFO:tasks.workunit.client.0.smithi046.stdout:created pool test snap snap5 2024-09-18T16:11:44.365 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2024-09-18T16:11:44.365 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2024-09-18T16:11:44.604 INFO:tasks.workunit.client.0.smithi046.stdout:created pool test snap snap6 2024-09-18T16:11:44.608 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.495962 bs=256 count=6 2024-09-18T16:11:44.610 INFO:tasks.workunit.client.0.smithi046.stderr:6+0 records in 2024-09-18T16:11:44.610 INFO:tasks.workunit.client.0.smithi046.stderr:6+0 records out 2024-09-18T16:11:44.610 INFO:tasks.workunit.client.0.smithi046.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 0.000302325 s, 5.1 MB/s 2024-09-18T16:11:44.610 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.495962 2024-09-18T16:11:44.873 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2024-09-18T16:11:44.873 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2024-09-18T16:11:45.094 INFO:tasks.workunit.client.0.smithi046.stdout:created pool test snap snap7 2024-09-18T16:11:45.098 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2024-09-18T16:11:45.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2024-09-18T16:11:45.483 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2024-09-18T16:11:45.583 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2024-09-18T16:11:45.584 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:11:45.584 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:11:45.585 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:11:45.585 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:11:45.585 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:11:45.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:11:45.899 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj1 2024-09-18T16:11:47.699 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:11:47.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2024-09-18T16:11:48.632 INFO:tasks.workunit.client.0.smithi046.stdout:WARNING: only removing head with clones present 2024-09-18T16:11:48.632 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:ff7b1f36:::obj1:head# 2024-09-18T16:11:49.181 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2024-09-18T16:11:49.181 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2024-09-18T16:11:50.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:11:50.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-18T16:11:51.420 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:c52c9666:::obj5:2# 2024-09-18T16:11:51.957 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2024-09-18T16:11:51.957 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2024-09-18T16:11:53.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:11:53.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:11:53.409 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-09-18T16:11:54.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-18T16:11:54.724 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:11:54.724 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-18T16:11:54.724 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:11:54.724 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-18T16:11:54.725 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-18T16:11:54.725 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-18T16:11:54.725 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-18T16:11:54.725 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-18T16:11:54.725 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-18T16:11:54.725 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-18T16:11:54.725 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:11:54.725 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-18T16:11:54.725 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-18T16:11:54.725 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-18T16:11:54.725 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-18T16:11:54.725 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:11:54.726 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-18T16:11:54.726 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:11:54.726 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:11:54.726 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:11:54.726 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-18T16:11:54.726 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:11:54.726 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-18T16:11:54.726 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:11:54.726 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:11:54.726 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:11:54.726 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-18T16:11:54.726 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:11:54.727 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:11:54.727 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:11:54.727 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-18T16:11:54.727 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:11:54.727 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:11:54.727 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:11:54.727 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-18T16:11:54.727 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:11:54.727 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:11:54.727 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:11:54.727 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:11:54.728 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-18T16:11:54.728 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-18T16:11:54.728 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-18T16:11:55.511 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:c52c9666:::obj5:1# 2024-09-18T16:11:56.047 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-09-18T16:11:57.362 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-18T16:11:57.364 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:11:57.364 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-18T16:11:57.365 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:11:57.365 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-18T16:11:57.365 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-18T16:11:57.365 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-18T16:11:57.365 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-18T16:11:57.365 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-18T16:11:57.365 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-18T16:11:57.365 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-18T16:11:57.365 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:11:57.365 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-18T16:11:57.365 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-18T16:11:57.365 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-18T16:11:57.365 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-18T16:11:57.365 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:11:57.366 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-18T16:11:57.366 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:11:57.366 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:11:57.366 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:11:57.366 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-18T16:11:57.366 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:11:57.366 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-18T16:11:57.366 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:11:57.366 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:11:57.366 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:11:57.366 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-18T16:11:57.366 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:11:57.366 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:11:57.366 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:11:57.366 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-18T16:11:57.367 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:11:57.367 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:11:57.367 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:11:57.367 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-18T16:11:57.367 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:11:57.367 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:11:57.367 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:11:57.367 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:11:57.367 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-18T16:11:57.367 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-18T16:11:57.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-09-18T16:11:57.370 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2024-09-18T16:11:57.370 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2024-09-18T16:11:58.703 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:11:58.704 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.495962 bs=256 count=18 2024-09-18T16:11:58.705 INFO:tasks.workunit.client.0.smithi046.stderr:18+0 records in 2024-09-18T16:11:58.705 INFO:tasks.workunit.client.0.smithi046.stderr:18+0 records out 2024-09-18T16:11:58.705 INFO:tasks.workunit.client.0.smithi046.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000383081 s, 12.0 MB/s 2024-09-18T16:11:58.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.495962 2024-09-18T16:12:00.044 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj3 2024-09-18T16:12:01.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:12:01.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.495962 bs=256 count=15 2024-09-18T16:12:01.693 INFO:tasks.workunit.client.0.smithi046.stderr:15+0 records in 2024-09-18T16:12:01.693 INFO:tasks.workunit.client.0.smithi046.stderr:15+0 records out 2024-09-18T16:12:01.693 INFO:tasks.workunit.client.0.smithi046.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000341237 s, 11.3 MB/s 2024-09-18T16:12:01.694 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.495962 2024-09-18T16:12:03.037 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj4 2024-09-18T16:12:03.037 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2024-09-18T16:12:04.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:12:04.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-18T16:12:05.352 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:0ee9ae15:::obj4:7# 2024-09-18T16:12:05.891 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-09-18T16:12:07.206 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-18T16:12:07.208 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-18T16:12:07.208 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:12:07.208 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-18T16:12:07.209 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-18T16:12:07.209 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-18T16:12:07.209 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-18T16:12:07.209 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-18T16:12:07.209 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-18T16:12:07.209 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-18T16:12:07.209 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:12:07.209 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-18T16:12:07.209 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-18T16:12:07.209 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-18T16:12:07.209 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-18T16:12:07.209 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-18T16:12:07.209 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:12:07.210 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:12:07.210 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-18T16:12:07.210 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:12:07.210 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-18T16:12:07.210 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:12:07.210 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:12:07.210 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-18T16:12:07.210 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:12:07.210 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:12:07.210 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-18T16:12:07.210 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:12:07.210 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:12:07.210 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-18T16:12:07.210 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:12:07.211 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:12:07.211 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:12:07.211 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-18T16:12:07.211 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:12:07.211 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:12:07.211 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:12:07.211 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:12:07.211 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:12:07.212 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:12:07.212 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:12:07.212 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj16 2024-09-18T16:12:07.212 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2024-09-18T16:12:08.584 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:12:08.584 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-18T16:12:09.375 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:461f8b5e:::obj16:7# 2024-09-18T16:12:09.912 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-09-18T16:12:11.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-18T16:12:11.476 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-18T16:12:11.476 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-18T16:12:11.477 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-18T16:12:11.477 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-18T16:12:11.477 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-18T16:12:11.477 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-18T16:12:11.477 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-18T16:12:11.478 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-18T16:12:11.478 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:12:11.478 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-18T16:12:11.478 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-18T16:12:11.478 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-18T16:12:11.478 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-18T16:12:11.479 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-18T16:12:11.479 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:12:11.479 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-18T16:12:11.479 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-18T16:12:11.479 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:12:11.479 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-18T16:12:11.480 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:12:11.480 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-18T16:12:11.480 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:12:11.480 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-18T16:12:11.480 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:12:11.480 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:12:11.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-18T16:12:11.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-09-18T16:12:11.482 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj2 2024-09-18T16:12:12.767 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:12:12.768 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2024-09-18T16:12:14.100 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:12:14.101 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2024-09-18T16:12:14.104 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:12:14.104 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.495962 bs=256 count=7 2024-09-18T16:12:14.105 INFO:tasks.workunit.client.0.smithi046.stderr:7+0 records in 2024-09-18T16:12:14.106 INFO:tasks.workunit.client.0.smithi046.stderr:7+0 records out 2024-09-18T16:12:14.106 INFO:tasks.workunit.client.0.smithi046.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 0.000284054 s, 6.3 MB/s 2024-09-18T16:12:14.106 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.495962 2024-09-18T16:12:15.661 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj6 2024-09-18T16:12:16.506 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:12:17.043 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:12:17.043 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2024-09-18T16:12:18.420 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj7 2024-09-18T16:12:19.299 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:12:19.835 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:12:19.836 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2024-09-18T16:12:21.124 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj8 2024-09-18T16:12:22.030 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:12:22.563 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:12:22.563 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2024-09-18T16:12:23.912 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj9 2024-09-18T16:12:24.737 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:12:25.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:12:25.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2024-09-18T16:12:26.724 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj10 2024-09-18T16:12:27.541 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:12:28.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:12:28.076 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2024-09-18T16:12:29.368 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj11 2024-09-18T16:12:30.237 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:12:30.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:12:30.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2024-09-18T16:12:32.084 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj12 2024-09-18T16:12:32.933 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:12:33.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:12:33.476 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2024-09-18T16:12:34.812 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj13 2024-09-18T16:12:35.681 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:12:36.467 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:12:36.468 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2024-09-18T16:12:37.856 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj14 2024-09-18T16:12:38.695 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:12:39.231 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:12:39.232 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2024-09-18T16:12:40.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2024-09-18T16:12:40.628 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj15 2024-09-18T16:12:41.496 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:12:42.039 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:12:42.039 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2024-09-18T16:12:43.367 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2024-09-18T16:12:43.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2024-09-18T16:12:43.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: rm -f testdata.495962 2024-09-18T16:12:43.372 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: expr 2 - 1 2024-09-18T16:12:43.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: seq 0 1 2024-09-18T16:12:43.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:12:43.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 0 2024-09-18T16:12:43.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-snaps 2024-09-18T16:12:43.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T16:12:43.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T16:12:43.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T16:12:43.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-snaps/0 2024-09-18T16:12:43.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3' 2024-09-18T16:12:43.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:12:43.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:12:43.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:12:43.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-09-18T16:12:43.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-09-18T16:12:43.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T16:12:43.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T16:12:43.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-18T16:12:43.378 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T16:12:43.378 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:12:43.378 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:12:43.379 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:12:43.379 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:12:43.379 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:12:43.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.495962/$cluster-$name.asok' 2024-09-18T16:12:43.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' 2024-09-18T16:12:43.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:12:43.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-18T16:12:43.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-18T16:12:43.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:12:43.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:12:43.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:12:43.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:12:43.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T16:12:43.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T16:12:43.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-snaps/0 2024-09-18T16:12:43.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T16:12:43.381 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T16:12:43.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:12:43.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-snaps/0/whoami 2024-09-18T16:12:43.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T16:12:43.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T16:12:43.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T16:12:43.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T16:12:43.416 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:12:43.412+0000 7f2edd7e1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:12:43.420 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:12:43.416+0000 7f2edd7e1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:12:43.422 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:12:43.420+0000 7f2edd7e1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:12:43.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T16:12:43.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:12:43.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:12:43.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:12:43.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:12:43.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:43.690 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:12:43.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:12:43.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:43.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:12:43.955 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:12:43.952+0000 7f2edd7e1540 -1 Falling back to public interface 2024-09-18T16:12:44.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:12:45.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:12:45.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:45.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:12:45.003 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:12:45.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:45.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:12:45.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:12:46.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:12:46.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:46.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:12:46.322 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:12:46.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:46.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:12:46.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:12:47.330 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:12:47.327+0000 7f2edd7e1540 -1 osd.0 23 log_to_monitors true 2024-09-18T16:12:47.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:12:47.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:47.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:12:47.638 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:12:47.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:47.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:12:48.013 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 24 last_clean_interval [5,23) [v2:127.0.0.1:6802/537411,v1:127.0.0.1:6803/537411] [v2:127.0.0.1:6804/537411,v1:127.0.0.1:6805/537411] exists,up a991ad25-2dc7-4b32-8aca-6d9121ad4e4f 2024-09-18T16:12:48.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:12:48.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:12:48.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:12:48.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:12:48.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 1 2024-09-18T16:12:48.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-snaps 2024-09-18T16:12:48.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T16:12:48.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T16:12:48.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T16:12:48.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-snaps/1 2024-09-18T16:12:48.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3' 2024-09-18T16:12:48.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:12:48.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:12:48.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:12:48.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-09-18T16:12:48.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-09-18T16:12:48.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T16:12:48.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T16:12:48.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-18T16:12:48.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T16:12:48.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:12:48.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:12:48.017 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:12:48.017 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:12:48.017 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:12:48.019 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.495962/$cluster-$name.asok' 2024-09-18T16:12:48.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' 2024-09-18T16:12:48.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:12:48.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-18T16:12:48.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-18T16:12:48.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:12:48.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:12:48.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:12:48.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:12:48.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T16:12:48.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T16:12:48.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-snaps/1 2024-09-18T16:12:48.020 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T16:12:48.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T16:12:48.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:12:48.021 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-snaps/1/whoami 2024-09-18T16:12:48.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T16:12:48.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T16:12:48.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T16:12:48.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T16:12:48.050 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:12:48.047+0000 7f391f8c4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:12:48.108 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:12:48.103+0000 7f391f8c4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:12:48.109 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:12:48.103+0000 7f391f8c4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:12:48.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T16:12:48.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:12:48.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T16:12:48.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:12:48.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:12:48.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:48.324 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:12:48.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:12:48.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:48.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:12:48.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:12:48.656 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:12:48.651+0000 7f391f8c4540 -1 Falling back to public interface 2024-09-18T16:12:49.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:12:49.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:49.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:12:49.633 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:12:49.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:49.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:12:49.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:12:50.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:12:50.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:50.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:12:50.954 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:12:50.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:50.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:12:51.281 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:12:51.275+0000 7f391f8c4540 -1 osd.1 23 log_to_monitors true 2024-09-18T16:12:51.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:12:52.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:12:52.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:52.284 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:12:52.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:12:52.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:52.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:12:52.598 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 29 up_thru 29 down_at 24 last_clean_interval [10,23) [v2:127.0.0.1:6810/537992,v1:127.0.0.1:6811/537992] [v2:127.0.0.1:6812/537992,v1:127.0.0.1:6813/537992] exists,up 45b5bd55-b062-42d9-a2f7-656472781481 2024-09-18T16:12:52.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:12:52.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:12:52.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:12:52.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: wait_for_clean 2024-09-18T16:12:52.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:12:52.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:12:52.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:12:52.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:12:52.601 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:12:52.603 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:12:52.603 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:12:52.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:12:52.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:12:52.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:12:52.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:12:52.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:12:52.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:12:52.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:12:52.815 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:12:53.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:12:53.125 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T16:12:53.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:12:53.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:12:53.125 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:12:53.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=111669149699 2024-09-18T16:12:53.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 111669149699 2024-09-18T16:12:53.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-111669149699' 2024-09-18T16:12:53.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:12:53.290 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:12:53.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051586 2024-09-18T16:12:53.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051586 2024-09-18T16:12:53.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-111669149699 1-124554051586' 2024-09-18T16:12:53.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:12:53.437 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-111669149699 2024-09-18T16:12:53.437 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:12:53.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:12:53.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-111669149699 2024-09-18T16:12:53.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:12:53.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=111669149699 2024-09-18T16:12:53.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 111669149699' 2024-09-18T16:12:53.443 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 111669149699 2024-09-18T16:12:53.443 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:12:53.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 111669149698 -lt 111669149699 2024-09-18T16:12:53.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:12:54.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:12:54.763 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:12:55.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 111669149699 -lt 111669149699 2024-09-18T16:12:55.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:12:55.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-124554051586 2024-09-18T16:12:55.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:12:55.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:12:55.084 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-124554051586 2024-09-18T16:12:55.084 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:12:55.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051586 2024-09-18T16:12:55.087 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 124554051586 2024-09-18T16:12:55.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2024-09-18T16:12:55.087 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:12:55.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051586 -lt 124554051586 2024-09-18T16:12:55.397 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:12:55.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:12:55.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:12:55.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:12:55.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:12:55.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:12:55.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:12:55.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:12:55.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:12:55.814 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:12:55.814 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:12:56.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:12:56.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:12:56.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:12:56.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:12:56.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:12:56.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:12:56.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:12:56.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:764: _scrub_snaps_multi: local pgid=1.0 2024-09-18T16:12:56.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: pg_scrub 1.0 2024-09-18T16:12:56.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: pg_scrub: local pgid=1.0 2024-09-18T16:12:56.526 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: pg_scrub: get_last_scrub_stamp 1.0 2024-09-18T16:12:56.526 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:12:56.526 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:12:56.526 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:12:56.527 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:12:56.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: pg_scrub: local last_scrub=2024-09-18T16:11:31.784724+0000 2024-09-18T16:12:56.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_scrub: ceph pg scrub 1.0 2024-09-18T16:12:57.174 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 1.0 on osd.1 to scrub 2024-09-18T16:12:57.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_scrub: wait_for_scrub 1.0 2024-09-18T16:11:31.784724+0000 2024-09-18T16:12:57.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-18T16:12:57.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T16:11:31.784724+0000 2024-09-18T16:12:57.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T16:12:57.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T16:12:57.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:12:57.184 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:12:57.184 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:12:57.184 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:12:57.184 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:12:57.185 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:12:57.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:11:31.784724+0000 '>' 2024-09-18T16:11:31.784724+0000 2024-09-18T16:12:57.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:12:58.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:12:58.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:12:58.488 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:12:58.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:12:58.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:12:58.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:12:58.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:12:58.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:11:31.784724+0000 '>' 2024-09-18T16:11:31.784724+0000 2024-09-18T16:12:58.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:12:59.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:12:59.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:12:59.804 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:12:59.804 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:12:59.804 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:12:59.804 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:12:59.804 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:13:00.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:11:31.784724+0000 '>' 2024-09-18T16:11:31.784724+0000 2024-09-18T16:13:00.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:13:01.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:13:01.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:13:01.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:13:01.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:13:01.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:13:01.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:13:01.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:13:01.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:11:31.784724+0000 '>' 2024-09-18T16:11:31.784724+0000 2024-09-18T16:13:01.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:13:02.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:13:02.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:13:02.444 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:13:02.444 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:13:02.444 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:13:02.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:13:02.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:13:02.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:11:31.784724+0000 '>' 2024-09-18T16:11:31.784724+0000 2024-09-18T16:13:02.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:13:03.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:13:03.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:13:03.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:13:03.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:13:03.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:13:03.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:13:03.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:13:04.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:12:58.846083+0000 '>' 2024-09-18T16:11:31.784724+0000 2024-09-18T16:13:04.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T16:13:04.088 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.1.log 2024-09-18T16:13:04.088 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: wc -l 2024-09-18T16:13:04.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: test 14 -gt 3 2024-09-18T16:13:04.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:770: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:13:04.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:770: _scrub_snaps_multi: wc -l 2024-09-18T16:13:04.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:770: _scrub_snaps_multi: test 14 -gt 3 2024-09-18T16:13:04.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:772: _scrub_snaps_multi: rados list-inconsistent-pg test 2024-09-18T16:13:04.139 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:774: _scrub_snaps_multi: jq '. | length' td/osd-scrub-snaps/json 2024-09-18T16:13:04.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:774: _scrub_snaps_multi: test 1 = 1 2024-09-18T16:13:04.179 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:776: _scrub_snaps_multi: jq -r '.[0]' td/osd-scrub-snaps/json 2024-09-18T16:13:04.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:776: _scrub_snaps_multi: test 1.0 = 1.0 2024-09-18T16:13:04.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:778: _scrub_snaps_multi: rados list-inconsistent-obj 1.0 --format=json-pretty 2024-09-18T16:13:04.263 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:13:04.263 INFO:tasks.workunit.client.0.smithi046.stdout: "epoch": 29, 2024-09-18T16:13:04.263 INFO:tasks.workunit.client.0.smithi046.stdout: "inconsistents": [ 2024-09-18T16:13:04.263 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.263 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:13:04.263 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj5", 2024-09-18T16:13:04.263 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:13:04.263 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:13:04.263 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:13:04.263 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 5 2024-09-18T16:13:04.264 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.264 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.264 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [ 2024-09-18T16:13:04.264 INFO:tasks.workunit.client.0.smithi046.stdout: "missing" 2024-09-18T16:13:04.264 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.264 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:13:04.264 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:13:04.264 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj5", 2024-09-18T16:13:04.264 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:13:04.264 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": 1, 2024-09-18T16:13:04.264 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 1718170787, 2024-09-18T16:13:04.264 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:13:04.265 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:13:04.265 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:13:04.265 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.265 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "17'19", 2024-09-18T16:13:04.265 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "16'5", 2024-09-18T16:13:04.265 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4241.0:1", 2024-09-18T16:13:04.265 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 5, 2024-09-18T16:13:04.265 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:13:04.265 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:11:37.866506+0000", 2024-09-18T16:13:04.265 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:11:37.867522+0000", 2024-09-18T16:13:04.265 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:13:04.265 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:13:04.266 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:13:04.266 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:13:04.266 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.266 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:13:04.266 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:13:04.266 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0xb7760be6", 2024-09-18T16:13:04.266 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:13:04.266 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:13:04.266 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:13:04.266 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:13:04.266 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:13:04.266 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:13:04.266 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.267 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:13:04.267 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.267 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:13:04.267 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.267 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:13:04.267 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:13:04.267 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.267 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032 2024-09-18T16:13:04.267 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.267 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.267 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:13:04.267 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:13:04.267 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:13:04.267 INFO:tasks.workunit.client.0.smithi046.stdout: "missing" 2024-09-18T16:13:04.268 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.268 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.268 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.268 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.268 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.268 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:13:04.268 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj5", 2024-09-18T16:13:04.268 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:13:04.268 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:13:04.268 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 2, 2024-09-18T16:13:04.268 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 20 2024-09-18T16:13:04.268 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.268 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.268 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [ 2024-09-18T16:13:04.269 INFO:tasks.workunit.client.0.smithi046.stdout: "missing" 2024-09-18T16:13:04.269 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.269 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:13:04.269 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:13:04.269 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj5", 2024-09-18T16:13:04.269 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:13:04.269 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": 2, 2024-09-18T16:13:04.269 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 1718170787, 2024-09-18T16:13:04.269 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:13:04.269 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:13:04.269 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:13:04.270 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.270 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "18'41", 2024-09-18T16:13:04.270 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "17'20", 2024-09-18T16:13:04.270 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4289.0:1", 2024-09-18T16:13:04.270 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 20, 2024-09-18T16:13:04.270 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:13:04.270 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:11:41.205025+0000", 2024-09-18T16:13:04.270 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:11:41.206627+0000", 2024-09-18T16:13:04.271 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:13:04.271 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:13:04.271 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:13:04.271 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:13:04.271 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.271 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:13:04.271 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:13:04.271 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0xcf1729b1", 2024-09-18T16:13:04.271 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:13:04.271 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:13:04.271 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:13:04.271 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:13:04.272 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:13:04.272 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:13:04.272 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.272 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:13:04.272 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.272 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:13:04.272 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.272 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:13:04.272 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:13:04.272 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.272 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256 2024-09-18T16:13:04.272 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.272 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.273 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:13:04.273 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:13:04.273 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:13:04.273 INFO:tasks.workunit.client.0.smithi046.stdout: "missing" 2024-09-18T16:13:04.273 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.273 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.273 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.273 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.273 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.273 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:13:04.273 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj5", 2024-09-18T16:13:04.273 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:13:04.273 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:13:04.274 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 4, 2024-09-18T16:13:04.274 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 42 2024-09-18T16:13:04.274 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.274 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:13:04.274 INFO:tasks.workunit.client.0.smithi046.stdout: "size_mismatch" 2024-09-18T16:13:04.274 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.274 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [ 2024-09-18T16:13:04.274 INFO:tasks.workunit.client.0.smithi046.stdout: "size_mismatch_info", 2024-09-18T16:13:04.274 INFO:tasks.workunit.client.0.smithi046.stdout: "obj_size_info_mismatch" 2024-09-18T16:13:04.274 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.274 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:13:04.274 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:13:04.274 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj5", 2024-09-18T16:13:04.275 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:13:04.275 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": 4, 2024-09-18T16:13:04.275 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 1718170787, 2024-09-18T16:13:04.275 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:13:04.275 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:13:04.275 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:13:04.275 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.275 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "20'45", 2024-09-18T16:13:04.275 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "18'42", 2024-09-18T16:13:04.275 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4325.0:1", 2024-09-18T16:13:04.276 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 42, 2024-09-18T16:13:04.276 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 512, 2024-09-18T16:13:04.276 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:11:42.889500+0000", 2024-09-18T16:13:04.276 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:11:42.907688+0000", 2024-09-18T16:13:04.276 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:13:04.276 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:13:04.276 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:13:04.276 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:13:04.276 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.276 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:13:04.276 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:13:04.276 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0x934f7396", 2024-09-18T16:13:04.277 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:13:04.277 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:13:04.277 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:13:04.277 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:13:04.277 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:13:04.277 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:13:04.277 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.277 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:13:04.277 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.277 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:13:04.277 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.277 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:13:04.277 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:13:04.278 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.278 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 512 2024-09-18T16:13:04.278 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.278 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.278 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:13:04.278 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:13:04.278 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:13:04.278 INFO:tasks.workunit.client.0.smithi046.stdout: "size_mismatch_info", 2024-09-18T16:13:04.278 INFO:tasks.workunit.client.0.smithi046.stdout: "obj_size_info_mismatch" 2024-09-18T16:13:04.278 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.278 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 4608, 2024-09-18T16:13:04.278 INFO:tasks.workunit.client.0.smithi046.stdout: "object_info": { 2024-09-18T16:13:04.279 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:13:04.279 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj5", 2024-09-18T16:13:04.279 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:13:04.279 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": 4, 2024-09-18T16:13:04.279 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 1718170787, 2024-09-18T16:13:04.279 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:13:04.279 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:13:04.279 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:13:04.279 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.279 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "20'45", 2024-09-18T16:13:04.279 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "18'42", 2024-09-18T16:13:04.279 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4325.0:1", 2024-09-18T16:13:04.279 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 42, 2024-09-18T16:13:04.279 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 512, 2024-09-18T16:13:04.280 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:11:42.889500+0000", 2024-09-18T16:13:04.280 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:11:42.907688+0000", 2024-09-18T16:13:04.280 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:13:04.280 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:13:04.280 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:13:04.280 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:13:04.280 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.280 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:13:04.280 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:13:04.280 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0x934f7396", 2024-09-18T16:13:04.280 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:13:04.280 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:13:04.280 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:13:04.281 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:13:04.281 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:13:04.281 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:13:04.281 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.281 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:13:04.281 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.281 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.281 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.281 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.281 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.281 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:13:04.281 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj4", 2024-09-18T16:13:04.281 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:13:04.281 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:13:04.282 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 7, 2024-09-18T16:13:04.282 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 4 2024-09-18T16:13:04.282 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.282 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.282 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [ 2024-09-18T16:13:04.282 INFO:tasks.workunit.client.0.smithi046.stdout: "missing" 2024-09-18T16:13:04.282 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.282 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:13:04.282 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:13:04.282 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj4", 2024-09-18T16:13:04.282 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:13:04.282 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": 7, 2024-09-18T16:13:04.282 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 2826278768, 2024-09-18T16:13:04.282 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:13:04.282 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:13:04.283 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:13:04.283 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.283 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "23'51", 2024-09-18T16:13:04.283 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "16'4", 2024-09-18T16:13:04.283 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4238.0:1", 2024-09-18T16:13:04.283 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 4, 2024-09-18T16:13:04.283 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:13:04.283 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:11:37.716151+0000", 2024-09-18T16:13:04.283 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:11:37.717584+0000", 2024-09-18T16:13:04.283 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:13:04.283 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:13:04.283 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:13:04.283 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:13:04.283 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.284 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:13:04.284 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:13:04.284 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0xb7760be6", 2024-09-18T16:13:04.284 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:13:04.284 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:13:04.284 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:13:04.284 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:13:04.284 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:13:04.284 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:13:04.284 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.284 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:13:04.284 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.284 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:13:04.285 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.285 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:13:04.285 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:13:04.285 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.285 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032 2024-09-18T16:13:04.285 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.285 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.285 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:13:04.285 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:13:04.285 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:13:04.285 INFO:tasks.workunit.client.0.smithi046.stdout: "missing" 2024-09-18T16:13:04.285 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.285 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.285 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.286 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.286 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.286 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:13:04.286 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj5", 2024-09-18T16:13:04.286 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:13:04.286 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:13:04.286 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 7, 2024-09-18T16:13:04.286 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 0 2024-09-18T16:13:04.286 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.286 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.286 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [ 2024-09-18T16:13:04.286 INFO:tasks.workunit.client.0.smithi046.stdout: "missing", 2024-09-18T16:13:04.286 INFO:tasks.workunit.client.0.smithi046.stdout: "info_missing" 2024-09-18T16:13:04.286 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.286 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:13:04.287 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.287 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:13:04.287 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:13:04.287 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:13:04.287 INFO:tasks.workunit.client.0.smithi046.stdout: "missing" 2024-09-18T16:13:04.287 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.287 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.287 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.287 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:13:04.287 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:13:04.287 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:13:04.287 INFO:tasks.workunit.client.0.smithi046.stdout: "info_missing" 2024-09-18T16:13:04.287 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.287 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1792 2024-09-18T16:13:04.288 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.288 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.288 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.288 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.288 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:13:04.288 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj1", 2024-09-18T16:13:04.288 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:13:04.288 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:13:04.288 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": "head", 2024-09-18T16:13:04.288 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 18 2024-09-18T16:13:04.288 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.288 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.288 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [ 2024-09-18T16:13:04.288 INFO:tasks.workunit.client.0.smithi046.stdout: "missing" 2024-09-18T16:13:04.289 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.289 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:13:04.289 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:13:04.289 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj1", 2024-09-18T16:13:04.289 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:13:04.289 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": -2, 2024-09-18T16:13:04.289 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 1828249343, 2024-09-18T16:13:04.289 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:13:04.289 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:13:04.289 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:13:04.289 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.289 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "17'18", 2024-09-18T16:13:04.289 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "16'1", 2024-09-18T16:13:04.290 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4286.0:1", 2024-09-18T16:13:04.290 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 18, 2024-09-18T16:13:04.290 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:13:04.290 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:11:40.949068+0000", 2024-09-18T16:13:04.290 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:11:41.033301+0000", 2024-09-18T16:13:04.290 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:13:04.290 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:13:04.290 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:13:04.290 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:13:04.290 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.290 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:13:04.290 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:13:04.290 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0xcf1729b1", 2024-09-18T16:13:04.290 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:13:04.291 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:13:04.291 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:13:04.291 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:13:04.291 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:13:04.291 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:13:04.291 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.291 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:13:04.291 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.291 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:13:04.291 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.291 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:13:04.291 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:13:04.291 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.291 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256 2024-09-18T16:13:04.291 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.292 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.292 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:13:04.292 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:13:04.292 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:13:04.292 INFO:tasks.workunit.client.0.smithi046.stdout: "missing" 2024-09-18T16:13:04.292 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.292 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.292 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.292 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.292 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.292 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:13:04.292 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj10", 2024-09-18T16:13:04.292 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:13:04.292 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:13:04.293 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": "head", 2024-09-18T16:13:04.293 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 32 2024-09-18T16:13:04.293 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.293 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:13:04.293 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset_inconsistency" 2024-09-18T16:13:04.293 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.293 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [], 2024-09-18T16:13:04.293 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:13:04.293 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:13:04.293 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj10", 2024-09-18T16:13:04.293 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:13:04.293 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": -2, 2024-09-18T16:13:04.293 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 718195851, 2024-09-18T16:13:04.294 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:13:04.294 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:13:04.294 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:13:04.294 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.294 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "17'32", 2024-09-18T16:13:04.294 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "16'10", 2024-09-18T16:13:04.294 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4307.0:1", 2024-09-18T16:13:04.294 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 32, 2024-09-18T16:13:04.294 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:13:04.294 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:11:42.056297+0000", 2024-09-18T16:13:04.294 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:11:42.057586+0000", 2024-09-18T16:13:04.294 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:13:04.294 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:13:04.295 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:13:04.295 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:13:04.295 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.295 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:13:04.295 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:13:04.295 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0xcf1729b1", 2024-09-18T16:13:04.295 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:13:04.295 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:13:04.295 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:13:04.295 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:13:04.295 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:13:04.295 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:13:04.295 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.296 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:13:04.296 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.296 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:13:04.296 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.296 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:13:04.296 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:13:04.296 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.296 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:13:04.296 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:13:04.296 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:13:04.296 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:13:04.296 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.296 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:13:04.297 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:13:04.297 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]", 2024-09-18T16:13:04.297 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:13:04.297 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T16:13:04.297 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.297 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.297 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.297 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.297 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.297 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.297 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:13:04.297 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:13:04.297 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.298 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:13:04.298 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:13:04.298 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:13:04.298 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:13:04.298 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.298 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:13:04.298 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:13:04.298 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "????", 2024-09-18T16:13:04.298 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:13:04.298 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T16:13:04.298 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.298 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.298 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.298 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.299 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.299 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.299 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.299 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.299 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:13:04.299 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj11", 2024-09-18T16:13:04.299 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:13:04.299 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:13:04.299 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": "head", 2024-09-18T16:13:04.299 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 34 2024-09-18T16:13:04.299 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.299 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:13:04.299 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset_inconsistency" 2024-09-18T16:13:04.299 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.300 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [], 2024-09-18T16:13:04.300 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:13:04.300 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:13:04.300 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj11", 2024-09-18T16:13:04.300 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:13:04.300 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": -2, 2024-09-18T16:13:04.300 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 693400951, 2024-09-18T16:13:04.300 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:13:04.300 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:13:04.300 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:13:04.300 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.300 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "17'34", 2024-09-18T16:13:04.300 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "16'11", 2024-09-18T16:13:04.300 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4310.0:1", 2024-09-18T16:13:04.301 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 34, 2024-09-18T16:13:04.301 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:13:04.301 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:11:42.172666+0000", 2024-09-18T16:13:04.301 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:11:42.174062+0000", 2024-09-18T16:13:04.301 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:13:04.301 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:13:04.301 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:13:04.301 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:13:04.301 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.301 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:13:04.301 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:13:04.301 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0xcf1729b1", 2024-09-18T16:13:04.301 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:13:04.302 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:13:04.302 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:13:04.302 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:13:04.302 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:13:04.302 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:13:04.302 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.302 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:13:04.302 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.302 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:13:04.302 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.302 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:13:04.302 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:13:04.302 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.303 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:13:04.303 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:13:04.303 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:13:04.303 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:13:04.303 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.303 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:13:04.303 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:13:04.303 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]", 2024-09-18T16:13:04.303 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:13:04.303 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T16:13:04.303 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.303 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.303 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.303 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.304 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.304 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.304 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:13:04.304 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:13:04.304 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.304 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:13:04.304 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:13:04.304 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:13:04.304 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [] 2024-09-18T16:13:04.304 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.304 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.304 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.304 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.304 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.305 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:13:04.305 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj13", 2024-09-18T16:13:04.305 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:13:04.305 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:13:04.305 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": "head", 2024-09-18T16:13:04.305 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 38 2024-09-18T16:13:04.305 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.305 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:13:04.305 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset_inconsistency" 2024-09-18T16:13:04.305 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.305 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [], 2024-09-18T16:13:04.305 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:13:04.305 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:13:04.305 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj13", 2024-09-18T16:13:04.306 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:13:04.306 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": -2, 2024-09-18T16:13:04.306 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 2087409765, 2024-09-18T16:13:04.306 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:13:04.306 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:13:04.306 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:13:04.306 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.306 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "17'38", 2024-09-18T16:13:04.306 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "16'13", 2024-09-18T16:13:04.306 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4316.0:1", 2024-09-18T16:13:04.306 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 38, 2024-09-18T16:13:04.306 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:13:04.306 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:11:42.433344+0000", 2024-09-18T16:13:04.306 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:11:42.434563+0000", 2024-09-18T16:13:04.307 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:13:04.307 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:13:04.307 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:13:04.307 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:13:04.307 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.307 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:13:04.307 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:13:04.307 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0xcf1729b1", 2024-09-18T16:13:04.307 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:13:04.307 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:13:04.307 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:13:04.307 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:13:04.307 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:13:04.307 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:13:04.308 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.308 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:13:04.308 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.308 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:13:04.308 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.308 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:13:04.308 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:13:04.308 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.308 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:13:04.308 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:13:04.308 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:13:04.308 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:13:04.308 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.309 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:13:04.309 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:13:04.309 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]", 2024-09-18T16:13:04.309 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:13:04.309 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T16:13:04.309 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.309 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.309 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.309 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.309 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.309 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.309 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:13:04.309 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:13:04.310 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.310 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:13:04.310 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:13:04.310 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:13:04.310 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:13:04.310 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.310 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:13:04.310 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:13:04.310 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]" 2024-09-18T16:13:04.310 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.310 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.310 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.310 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.310 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.311 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.311 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.311 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:13:04.311 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj14", 2024-09-18T16:13:04.311 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:13:04.311 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:13:04.311 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": "head", 2024-09-18T16:13:04.311 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 40 2024-09-18T16:13:04.311 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.311 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:13:04.311 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset_inconsistency" 2024-09-18T16:13:04.311 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.312 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [], 2024-09-18T16:13:04.312 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:13:04.312 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:13:04.312 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj14", 2024-09-18T16:13:04.312 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:13:04.312 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": -2, 2024-09-18T16:13:04.312 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 2484217095, 2024-09-18T16:13:04.312 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:13:04.312 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:13:04.312 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:13:04.312 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.312 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "17'40", 2024-09-18T16:13:04.312 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "16'14", 2024-09-18T16:13:04.312 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4319.0:1", 2024-09-18T16:13:04.313 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 40, 2024-09-18T16:13:04.313 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:13:04.313 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:11:42.567292+0000", 2024-09-18T16:13:04.313 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:11:42.568411+0000", 2024-09-18T16:13:04.313 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:13:04.313 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:13:04.313 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:13:04.313 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:13:04.313 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.313 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:13:04.313 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:13:04.313 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0xcf1729b1", 2024-09-18T16:13:04.313 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:13:04.314 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:13:04.314 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:13:04.314 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:13:04.314 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:13:04.314 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:13:04.314 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.314 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:13:04.314 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.314 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:13:04.314 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.314 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:13:04.314 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:13:04.314 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.314 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:13:04.315 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:13:04.315 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:13:04.315 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:13:04.315 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.315 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:13:04.315 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:13:04.315 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]", 2024-09-18T16:13:04.315 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:13:04.315 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T16:13:04.315 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.315 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.315 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.315 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.315 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.316 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.316 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:13:04.316 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:13:04.316 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.316 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:13:04.316 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:13:04.316 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:13:04.316 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:13:04.316 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.316 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:13:04.316 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1033, 2024-09-18T16:13:04.316 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]", 2024-09-18T16:13:04.316 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:13:04.316 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T16:13:04.317 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.317 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.317 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.317 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.317 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.317 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.317 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.317 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.317 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:13:04.317 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj15", 2024-09-18T16:13:04.317 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:13:04.317 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:13:04.317 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": "head", 2024-09-18T16:13:04.318 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 15 2024-09-18T16:13:04.318 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.318 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.318 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [ 2024-09-18T16:13:04.318 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset_corrupted" 2024-09-18T16:13:04.318 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.318 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:13:04.318 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:13:04.318 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj15", 2024-09-18T16:13:04.318 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:13:04.318 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": -2, 2024-09-18T16:13:04.318 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 612772309, 2024-09-18T16:13:04.318 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:13:04.318 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:13:04.318 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:13:04.319 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.319 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "16'15", 2024-09-18T16:13:04.319 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "0'0", 2024-09-18T16:13:04.319 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4271.0:1", 2024-09-18T16:13:04.319 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 15, 2024-09-18T16:13:04.319 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:13:04.319 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:11:39.373249+0000", 2024-09-18T16:13:04.319 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:11:39.374732+0000", 2024-09-18T16:13:04.319 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:13:04.319 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:13:04.319 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:13:04.319 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:13:04.319 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.319 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:13:04.319 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:13:04.320 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0xb7760be6", 2024-09-18T16:13:04.320 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:13:04.320 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:13:04.320 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:13:04.320 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:13:04.320 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:13:04.320 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:13:04.320 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.320 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:13:04.320 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.320 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:13:04.320 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.321 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:13:04.321 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:13:04.321 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.321 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:13:04.321 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:13:04.321 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 0, 2024-09-18T16:13:04.321 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [] 2024-09-18T16:13:04.321 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.322 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.322 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.322 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:13:04.322 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:13:04.322 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:13:04.322 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset_corrupted" 2024-09-18T16:13:04.322 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.322 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:13:04.322 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": "Z2FyYmFnZQo=" 2024-09-18T16:13:04.322 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.322 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.322 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.322 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.322 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:13:04.323 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj2", 2024-09-18T16:13:04.323 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:13:04.323 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:13:04.323 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": "head", 2024-09-18T16:13:04.323 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 56 2024-09-18T16:13:04.323 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.323 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.323 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [ 2024-09-18T16:13:04.323 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset_missing" 2024-09-18T16:13:04.323 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.323 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:13:04.323 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:13:04.323 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj2", 2024-09-18T16:13:04.324 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:13:04.324 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": -2, 2024-09-18T16:13:04.324 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 1058988552, 2024-09-18T16:13:04.324 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:13:04.324 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:13:04.324 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:13:04.324 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.324 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "23'56", 2024-09-18T16:13:04.324 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "20'48", 2024-09-18T16:13:04.324 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4361.0:1", 2024-09-18T16:13:04.324 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 56, 2024-09-18T16:13:04.324 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 0, 2024-09-18T16:13:04.324 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:11:45.518536+0000", 2024-09-18T16:13:04.324 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:11:45.519975+0000", 2024-09-18T16:13:04.325 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:13:04.325 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:13:04.325 INFO:tasks.workunit.client.0.smithi046.stdout: "whiteout", 2024-09-18T16:13:04.325 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty" 2024-09-18T16:13:04.325 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.325 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:13:04.325 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:13:04.325 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0xffffffff", 2024-09-18T16:13:04.325 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:13:04.325 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:13:04.325 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:13:04.325 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:13:04.325 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:13:04.325 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:13:04.326 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.326 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:13:04.326 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.326 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:13:04.326 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.326 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:13:04.326 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:13:04.326 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.326 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 0, 2024-09-18T16:13:04.326 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:13:04.326 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 7, 2024-09-18T16:13:04.326 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:13:04.326 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.326 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 4, 2024-09-18T16:13:04.326 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:13:04.327 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]", 2024-09-18T16:13:04.327 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:13:04.327 INFO:tasks.workunit.client.0.smithi046.stdout: 4, 2024-09-18T16:13:04.327 INFO:tasks.workunit.client.0.smithi046.stdout: 3, 2024-09-18T16:13:04.327 INFO:tasks.workunit.client.0.smithi046.stdout: 2, 2024-09-18T16:13:04.327 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T16:13:04.327 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.327 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.327 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.327 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 7, 2024-09-18T16:13:04.327 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1024, 2024-09-18T16:13:04.327 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]", 2024-09-18T16:13:04.327 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:13:04.327 INFO:tasks.workunit.client.0.smithi046.stdout: 7, 2024-09-18T16:13:04.327 INFO:tasks.workunit.client.0.smithi046.stdout: 6, 2024-09-18T16:13:04.328 INFO:tasks.workunit.client.0.smithi046.stdout: 5 2024-09-18T16:13:04.328 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.328 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.328 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.328 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.328 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.328 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.328 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:13:04.328 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:13:04.328 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:13:04.328 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset_missing" 2024-09-18T16:13:04.328 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.328 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 0 2024-09-18T16:13:04.328 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.328 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.329 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.329 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.329 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:13:04.329 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj3", 2024-09-18T16:13:04.329 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:13:04.329 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:13:04.329 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": "head", 2024-09-18T16:13:04.329 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 44 2024-09-18T16:13:04.330 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.330 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:13:04.330 INFO:tasks.workunit.client.0.smithi046.stdout: "size_mismatch" 2024-09-18T16:13:04.330 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.330 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [ 2024-09-18T16:13:04.330 INFO:tasks.workunit.client.0.smithi046.stdout: "size_mismatch_info", 2024-09-18T16:13:04.330 INFO:tasks.workunit.client.0.smithi046.stdout: "obj_size_info_mismatch" 2024-09-18T16:13:04.330 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.330 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:13:04.330 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:13:04.330 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj3", 2024-09-18T16:13:04.330 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:13:04.330 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": -2, 2024-09-18T16:13:04.330 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 1643547569, 2024-09-18T16:13:04.331 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:13:04.331 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:13:04.331 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:13:04.331 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.331 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "19'44", 2024-09-18T16:13:04.331 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "17'22", 2024-09-18T16:13:04.331 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4331.0:1", 2024-09-18T16:13:04.331 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 44, 2024-09-18T16:13:04.331 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 768, 2024-09-18T16:13:04.331 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:11:43.343484+0000", 2024-09-18T16:13:04.331 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:11:43.351176+0000", 2024-09-18T16:13:04.331 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:13:04.331 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:13:04.331 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:13:04.332 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:13:04.332 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.332 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:13:04.332 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:13:04.332 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0xc2f85ebe", 2024-09-18T16:13:04.332 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:13:04.332 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:13:04.332 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:13:04.332 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:13:04.332 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:13:04.332 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:13:04.332 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.332 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:13:04.332 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.333 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:13:04.333 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.333 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:13:04.333 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:13:04.333 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.333 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 768 2024-09-18T16:13:04.333 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.333 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.333 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:13:04.333 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:13:04.333 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:13:04.333 INFO:tasks.workunit.client.0.smithi046.stdout: "size_mismatch_info", 2024-09-18T16:13:04.333 INFO:tasks.workunit.client.0.smithi046.stdout: "obj_size_info_mismatch" 2024-09-18T16:13:04.334 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.334 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 3840, 2024-09-18T16:13:04.334 INFO:tasks.workunit.client.0.smithi046.stdout: "object_info": { 2024-09-18T16:13:04.334 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:13:04.334 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj3", 2024-09-18T16:13:04.334 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:13:04.334 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": -2, 2024-09-18T16:13:04.334 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 1643547569, 2024-09-18T16:13:04.334 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:13:04.334 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:13:04.334 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:13:04.334 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.334 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "19'44", 2024-09-18T16:13:04.334 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "17'22", 2024-09-18T16:13:04.335 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4331.0:1", 2024-09-18T16:13:04.335 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 44, 2024-09-18T16:13:04.335 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 768, 2024-09-18T16:13:04.335 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:11:43.343484+0000", 2024-09-18T16:13:04.335 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:11:43.351176+0000", 2024-09-18T16:13:04.335 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:13:04.335 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:13:04.335 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:13:04.335 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:13:04.335 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.335 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:13:04.335 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:13:04.335 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0xc2f85ebe", 2024-09-18T16:13:04.335 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:13:04.336 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:13:04.336 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:13:04.336 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:13:04.336 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:13:04.336 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:13:04.336 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.336 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:13:04.336 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.336 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.336 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.336 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.336 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.336 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:13:04.337 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj6", 2024-09-18T16:13:04.337 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:13:04.337 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:13:04.337 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": "head", 2024-09-18T16:13:04.337 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 24 2024-09-18T16:13:04.337 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.337 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:13:04.337 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset_inconsistency" 2024-09-18T16:13:04.337 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.337 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [], 2024-09-18T16:13:04.337 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:13:04.337 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:13:04.337 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj6", 2024-09-18T16:13:04.338 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:13:04.338 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": -2, 2024-09-18T16:13:04.338 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 2202164420, 2024-09-18T16:13:04.338 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:13:04.338 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:13:04.338 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:13:04.338 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.338 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "17'24", 2024-09-18T16:13:04.338 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "16'6", 2024-09-18T16:13:04.338 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4295.0:1", 2024-09-18T16:13:04.338 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 24, 2024-09-18T16:13:04.339 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:13:04.339 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:11:41.524640+0000", 2024-09-18T16:13:04.339 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:11:41.525793+0000", 2024-09-18T16:13:04.339 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:13:04.339 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:13:04.339 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:13:04.339 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:13:04.339 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.339 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:13:04.339 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:13:04.339 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0xcf1729b1", 2024-09-18T16:13:04.339 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:13:04.339 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:13:04.339 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:13:04.340 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:13:04.340 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:13:04.340 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:13:04.340 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.340 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:13:04.340 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.340 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:13:04.340 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.340 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:13:04.340 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:13:04.340 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.340 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:13:04.341 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:13:04.341 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:13:04.341 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:13:04.341 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.341 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:13:04.341 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:13:04.341 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]", 2024-09-18T16:13:04.341 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:13:04.341 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T16:13:04.341 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.341 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.341 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.341 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.342 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.342 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.342 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:13:04.342 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:13:04.342 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.342 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:13:04.342 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:13:04.342 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:13:04.342 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [] 2024-09-18T16:13:04.342 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.342 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.342 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.342 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.343 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.343 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:13:04.343 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj7", 2024-09-18T16:13:04.343 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:13:04.343 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:13:04.343 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": "head", 2024-09-18T16:13:04.343 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 26 2024-09-18T16:13:04.343 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.343 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:13:04.343 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset_inconsistency" 2024-09-18T16:13:04.343 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.343 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [], 2024-09-18T16:13:04.343 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:13:04.343 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:13:04.343 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj7", 2024-09-18T16:13:04.344 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:13:04.344 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": -2, 2024-09-18T16:13:04.344 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 1552453721, 2024-09-18T16:13:04.344 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:13:04.344 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:13:04.344 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:13:04.344 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.344 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "17'26", 2024-09-18T16:13:04.344 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "16'7", 2024-09-18T16:13:04.344 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4298.0:1", 2024-09-18T16:13:04.344 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 26, 2024-09-18T16:13:04.344 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:13:04.344 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:11:41.644155+0000", 2024-09-18T16:13:04.344 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:11:41.645650+0000", 2024-09-18T16:13:04.345 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:13:04.345 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:13:04.345 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:13:04.345 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:13:04.345 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.345 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:13:04.345 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:13:04.345 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0xcf1729b1", 2024-09-18T16:13:04.345 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:13:04.345 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:13:04.345 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:13:04.345 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:13:04.345 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:13:04.345 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:13:04.345 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.346 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:13:04.346 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.346 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:13:04.346 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.346 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:13:04.346 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:13:04.346 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.346 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:13:04.346 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:13:04.346 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:13:04.346 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:13:04.346 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.346 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:13:04.346 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:13:04.346 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]", 2024-09-18T16:13:04.347 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:13:04.347 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T16:13:04.347 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.347 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.347 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.347 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.347 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.347 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.347 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:13:04.347 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:13:04.347 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.347 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:13:04.347 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:13:04.347 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 0, 2024-09-18T16:13:04.348 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [] 2024-09-18T16:13:04.348 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.348 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.348 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.348 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.348 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.348 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:13:04.348 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj8", 2024-09-18T16:13:04.348 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:13:04.348 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:13:04.348 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": "head", 2024-09-18T16:13:04.348 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 28 2024-09-18T16:13:04.349 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.349 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:13:04.349 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset_inconsistency" 2024-09-18T16:13:04.349 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.349 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [], 2024-09-18T16:13:04.349 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:13:04.349 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:13:04.349 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj8", 2024-09-18T16:13:04.349 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:13:04.349 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": -2, 2024-09-18T16:13:04.349 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 2381834917, 2024-09-18T16:13:04.349 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:13:04.349 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:13:04.350 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:13:04.350 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.350 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "17'28", 2024-09-18T16:13:04.350 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "16'8", 2024-09-18T16:13:04.350 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4301.0:1", 2024-09-18T16:13:04.350 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 28, 2024-09-18T16:13:04.350 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:13:04.350 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:11:41.826333+0000", 2024-09-18T16:13:04.350 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:11:41.827836+0000", 2024-09-18T16:13:04.350 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:13:04.350 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:13:04.350 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:13:04.350 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:13:04.351 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.351 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:13:04.351 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:13:04.351 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0xcf1729b1", 2024-09-18T16:13:04.351 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:13:04.351 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:13:04.351 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:13:04.351 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:13:04.351 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:13:04.351 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:13:04.351 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.351 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:13:04.352 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.352 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:13:04.352 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.352 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:13:04.352 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:13:04.352 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.352 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:13:04.352 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:13:04.352 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:13:04.352 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:13:04.352 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.352 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:13:04.352 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:13:04.353 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]", 2024-09-18T16:13:04.353 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:13:04.353 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T16:13:04.353 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.353 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.353 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.353 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.353 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.353 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.353 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:13:04.353 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:13:04.353 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.353 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:13:04.353 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:13:04.353 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 0, 2024-09-18T16:13:04.354 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:13:04.354 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.354 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:13:04.354 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:13:04.354 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]", 2024-09-18T16:13:04.354 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:13:04.354 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T16:13:04.354 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.354 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.354 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.354 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.354 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.354 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.354 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.355 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.355 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:13:04.355 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj9", 2024-09-18T16:13:04.355 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:13:04.355 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:13:04.355 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": "head", 2024-09-18T16:13:04.355 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 30 2024-09-18T16:13:04.355 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.355 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:13:04.355 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset_inconsistency" 2024-09-18T16:13:04.355 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.355 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [], 2024-09-18T16:13:04.355 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:13:04.356 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:13:04.356 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj9", 2024-09-18T16:13:04.356 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:13:04.356 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": -2, 2024-09-18T16:13:04.356 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 3833113727, 2024-09-18T16:13:04.356 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:13:04.356 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:13:04.356 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:13:04.356 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.356 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "17'30", 2024-09-18T16:13:04.356 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "16'9", 2024-09-18T16:13:04.356 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4304.0:1", 2024-09-18T16:13:04.356 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 30, 2024-09-18T16:13:04.356 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:13:04.357 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:11:41.937882+0000", 2024-09-18T16:13:04.357 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:11:41.939013+0000", 2024-09-18T16:13:04.357 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:13:04.357 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:13:04.357 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:13:04.357 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:13:04.357 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:13:04.357 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:13:04.357 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:13:04.357 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0xcf1729b1", 2024-09-18T16:13:04.357 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:13:04.357 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:13:04.357 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:13:04.358 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:13:04.358 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:13:04.358 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:13:04.358 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.358 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:13:04.358 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.358 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:13:04.358 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.358 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:13:04.358 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:13:04.358 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.358 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:13:04.358 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:13:04.358 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:13:04.359 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:13:04.359 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.359 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:13:04.359 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:13:04.359 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]", 2024-09-18T16:13:04.359 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:13:04.359 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T16:13:04.360 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.360 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.360 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.360 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.360 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:13:04.360 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.360 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:13:04.360 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:13:04.360 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:13:04.360 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:13:04.360 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:13:04.360 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:13:04.360 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:13:04.360 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:13:04.361 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:13:04.361 INFO:tasks.workunit.client.0.smithi046.stdout: "size": "????", 2024-09-18T16:13:04.361 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]", 2024-09-18T16:13:04.361 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:13:04.361 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T16:13:04.361 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.361 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.361 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.361 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.361 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.361 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.361 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:13:04.361 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:13:04.361 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:13:04.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:780: _scrub_snaps_multi: rados list-inconsistent-snapset 1.0 2024-09-18T16:13:04.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:783: _scrub_snaps_multi: '[' primary = replica ']' 2024-09-18T16:13:04.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:794: _scrub_snaps_multi: scruberrors=30 2024-09-18T16:13:04.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:795: _scrub_snaps_multi: jq .inconsistents 2024-09-18T16:13:04.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:795: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-18T16:13:04.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1049: _scrub_snaps_multi: jq .inconsistents td/osd-scrub-snaps/json 2024-09-18T16:13:04.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1049: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-18T16:13:04.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1050: _scrub_snaps_multi: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-18T16:13:04.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-18T16:13:04.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1051: _scrub_snaps_multi: test no = yes 2024-09-18T16:13:04.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1056: _scrub_snaps_multi: test '' = yes 2024-09-18T16:13:04.399 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1061: _scrub_snaps_multi: find td/osd-scrub-snaps 2024-09-18T16:13:04.399 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1061: _scrub_snaps_multi: grep 'osd[^/]*\.pid' 2024-09-18T16:13:04.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1061: _scrub_snaps_multi: pidfiles='td/osd-scrub-snaps/osd.0.pid 2024-09-18T16:13:04.402 INFO:tasks.workunit.client.0.smithi046.stderr:td/osd-scrub-snaps/osd.1.pid' 2024-09-18T16:13:04.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1062: _scrub_snaps_multi: pids= 2024-09-18T16:13:04.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1063: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-09-18T16:13:04.402 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1065: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.0.pid 2024-09-18T16:13:04.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1065: _scrub_snaps_multi: pids+='537411 ' 2024-09-18T16:13:04.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1063: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-09-18T16:13:04.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1065: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.1.pid 2024-09-18T16:13:04.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1065: _scrub_snaps_multi: pids+='537992 ' 2024-09-18T16:13:04.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1068: _scrub_snaps_multi: ERRORS=0 2024-09-18T16:13:04.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1072: _scrub_snaps_multi: '[' primary = primary ']' 2024-09-18T16:13:04.406 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1074: _scrub_snaps_multi: seq 1 7 2024-09-18T16:13:04.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1074: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-18T16:13:04.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1076: _scrub_snaps_multi: rados -p test rmsnap snap1 2024-09-18T16:13:04.586 INFO:tasks.workunit.client.0.smithi046.stdout:removed pool test snap snap1 2024-09-18T16:13:04.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1074: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-18T16:13:04.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1076: _scrub_snaps_multi: rados -p test rmsnap snap2 2024-09-18T16:13:04.766 INFO:tasks.workunit.client.0.smithi046.stdout:removed pool test snap snap2 2024-09-18T16:13:04.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1074: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-18T16:13:04.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1076: _scrub_snaps_multi: rados -p test rmsnap snap3 2024-09-18T16:13:05.109 INFO:tasks.workunit.client.0.smithi046.stdout:removed pool test snap snap3 2024-09-18T16:13:05.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1074: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-18T16:13:05.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1076: _scrub_snaps_multi: rados -p test rmsnap snap4 2024-09-18T16:13:05.395 INFO:tasks.workunit.client.0.smithi046.stdout:removed pool test snap snap4 2024-09-18T16:13:05.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1074: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-18T16:13:05.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1076: _scrub_snaps_multi: rados -p test rmsnap snap5 2024-09-18T16:13:05.789 INFO:tasks.workunit.client.0.smithi046.stdout:removed pool test snap snap5 2024-09-18T16:13:05.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1074: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-18T16:13:05.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1076: _scrub_snaps_multi: rados -p test rmsnap snap6 2024-09-18T16:13:06.040 INFO:tasks.workunit.client.0.smithi046.stdout:removed pool test snap snap6 2024-09-18T16:13:06.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1074: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-18T16:13:06.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1076: _scrub_snaps_multi: rados -p test rmsnap snap7 2024-09-18T16:13:06.274 INFO:tasks.workunit.client.0.smithi046.stdout:removed pool test snap snap7 2024-09-18T16:13:06.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: sleep 5 2024-09-18T16:13:11.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: local -i loop=0 2024-09-18T16:13:11.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: ceph pg dump pgs 2024-09-18T16:13:11.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: grep -q snaptrim 2024-09-18T16:13:11.586 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:13:11.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: ceph pg dump pgs 2024-09-18T16:13:11.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: grep -q snaptrim_error 2024-09-18T16:13:11.937 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:13:11.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1084: _scrub_snaps_multi: break 2024-09-18T16:13:11.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1094: _scrub_snaps_multi: ceph pg dump pgs 2024-09-18T16:13:12.231 INFO:tasks.workunit.client.0.smithi046.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T16:13:12.231 INFO:tasks.workunit.client.0.smithi046.stdout:1.0 36 0 0 0 0 24448 0 0 56 0 56 active+clean+inconsistent+snaptrim_error 2024-09-18T16:13:06.307176+0000 23'56 37:112 [1,0] 1 [1,0] 1 23'56 2024-09-18T16:12:58.846083+0000 0'0 2024-09-18T16:11:31.784724+0000 7 1 periodic scrub scheduled @ 2024-09-20T00:29:48.967689+0000 33 0 2024-09-18T16:13:12.231 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-09-18T16:13:12.231 INFO:tasks.workunit.client.0.smithi046.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T16:13:12.234 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:13:12.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1096: _scrub_snaps_multi: for pid in $pids 2024-09-18T16:13:12.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1098: _scrub_snaps_multi: kill -0 537411 2024-09-18T16:13:12.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1096: _scrub_snaps_multi: for pid in $pids 2024-09-18T16:13:12.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1098: _scrub_snaps_multi: kill -0 537992 2024-09-18T16:13:12.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1105: _scrub_snaps_multi: kill_daemons td/osd-scrub-snaps 2024-09-18T16:13:12.244 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:13:12.244 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:13:12.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:13:12.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:13:12.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:13:17.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:13:17.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1107: _scrub_snaps_multi: declare -a err_strings 2024-09-18T16:13:17.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1108: _scrub_snaps_multi: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' 2024-09-18T16:13:17.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1109: _scrub_snaps_multi: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' 2024-09-18T16:13:17.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1110: _scrub_snaps_multi: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' 2024-09-18T16:13:17.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1111: _scrub_snaps_multi: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' 2024-09-18T16:13:17.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1112: _scrub_snaps_multi: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' 2024-09-18T16:13:17.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' 2024-09-18T16:13:17.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1114: _scrub_snaps_multi: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' 2024-09-18T16:13:17.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 30 errors' 2024-09-18T16:13:17.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1117: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-18T16:13:17.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1119: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' td/osd-scrub-snaps/osd.1.log 2024-09-18T16:13:17.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1117: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-18T16:13:17.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1119: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-09-18T16:13:17.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1117: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-18T16:13:17.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1119: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' td/osd-scrub-snaps/osd.1.log 2024-09-18T16:13:17.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1117: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-18T16:13:17.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1119: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' td/osd-scrub-snaps/osd.1.log 2024-09-18T16:13:17.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1117: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-18T16:13:17.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1119: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-09-18T16:13:17.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1117: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-18T16:13:17.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1119: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' td/osd-scrub-snaps/osd.1.log 2024-09-18T16:13:17.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1117: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-18T16:13:17.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1119: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' td/osd-scrub-snaps/osd.1.log 2024-09-18T16:13:17.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1117: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-18T16:13:17.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1119: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 30 errors' td/osd-scrub-snaps/osd.1.log 2024-09-18T16:13:17.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1127: _scrub_snaps_multi: declare -a rep_err_strings 2024-09-18T16:13:17.607 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1128: _scrub_snaps_multi: eval echo '$primary' 2024-09-18T16:13:17.607 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1128: _scrub_snaps_multi: echo 1 2024-09-18T16:13:17.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1128: _scrub_snaps_multi: osd=1 2024-09-18T16:13:17.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1129: _scrub_snaps_multi: rep_err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2024-09-18T16:13:17.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1130: _scrub_snaps_multi: for err_string in "${rep_err_strings[@]}" 2024-09-18T16:13:17.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1132: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.1.log 2024-09-18T16:13:17.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1139: _scrub_snaps_multi: '[' 0 '!=' 0 ']' 2024-09-18T16:13:17.613 INFO:tasks.workunit.client.0.smithi046.stdout:TEST PASSED 2024-09-18T16:13:17.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: echo 'TEST PASSED' 2024-09-18T16:13:17.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1146: _scrub_snaps_multi: return 0 2024-09-18T16:13:17.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1164: TEST_scrub_snaps_primary: err=0 2024-09-18T16:13:17.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1165: TEST_scrub_snaps_primary: CEPH_ARGS='--fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-09-18T16:13:17.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1166: TEST_scrub_snaps_primary: return 0 2024-09-18T16:13:17.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2024-09-18T16:13:17.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-09-18T16:13:17.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:13:17.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-18T16:13:17.616 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:13:17.616 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:13:17.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:13:17.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:13:17.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:13:17.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:13:17.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:13:17.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:13:17.629 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:13:17.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:13:17.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:13:17.631 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:13:17.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:13:17.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:13:17.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:13:17.634 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:13:17.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:13:17.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:13:17.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-09-18T16:13:17.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:13:17.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:13:17.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:13:17.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.495962 2024-09-18T16:13:17.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:13:17.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:13:17.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2024-09-18T16:13:17.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2024-09-18T16:13:17.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-snaps 2024-09-18T16:13:17.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-snaps 2024-09-18T16:13:17.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-09-18T16:13:17.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:13:17.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-18T16:13:17.660 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:13:17.660 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:13:17.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:13:17.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:13:17.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:13:17.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:13:17.665 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:13:17.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:13:17.667 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:13:17.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:13:17.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:13:17.670 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:13:17.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:13:17.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:13:17.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:13:17.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:13:17.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:13:17.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:13:17.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-09-18T16:13:17.678 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:13:17.678 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:13:17.678 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:13:17.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.495962 2024-09-18T16:13:17.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:13:17.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:13:17.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-snaps 2024-09-18T16:13:17.683 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T16:13:17.683 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:13:17.683 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:13:17.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.495962 2024-09-18T16:13:17.686 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T16:13:17.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T16:13:17.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T16:13:17.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2024-09-18T16:13:17.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps_replica td/osd-scrub-snaps 2024-09-18T16:13:17.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1150: TEST_scrub_snaps_replica: local dir=td/osd-scrub-snaps 2024-09-18T16:13:17.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1151: TEST_scrub_snaps_replica: ORIG_ARGS='--fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-09-18T16:13:17.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1152: TEST_scrub_snaps_replica: CEPH_ARGS+=' --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3' 2024-09-18T16:13:17.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1153: TEST_scrub_snaps_replica: _scrub_snaps_multi td/osd-scrub-snaps replica 2024-09-18T16:13:17.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:720: _scrub_snaps_multi: local dir=td/osd-scrub-snaps 2024-09-18T16:13:17.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:721: _scrub_snaps_multi: local poolname=test 2024-09-18T16:13:17.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:722: _scrub_snaps_multi: local OBJS=16 2024-09-18T16:13:17.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:723: _scrub_snaps_multi: local OSDS=2 2024-09-18T16:13:17.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:724: _scrub_snaps_multi: local which=replica 2024-09-18T16:13:17.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:726: _scrub_snaps_multi: TESTDATA=testdata.495962 2024-09-18T16:13:17.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:728: _scrub_snaps_multi: run_mon td/osd-scrub-snaps a --osd_pool_default_size=2 2024-09-18T16:13:17.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-snaps 2024-09-18T16:13:17.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T16:13:17.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T16:13:17.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T16:13:17.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-snaps/a 2024-09-18T16:13:17.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=2 2024-09-18T16:13:18.065 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T16:13:18.065 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:13:18.065 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:13:18.065 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:13:18.065 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:13:18.066 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:13:18.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.495962/$cluster-$name.asok' 2024-09-18T16:13:18.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-18T16:13:18.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T16:13:18.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T16:13:18.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:13:18.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:13:18.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T16:13:18.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T16:13:18.374 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:13:18.374 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:13:18.374 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:13:18.375 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:13:18.375 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:13:18.375 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:13:18.375 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.495962/ceph-mon.a.asok 2024-09-18T16:13:18.375 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:13:18.376 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.495962/ceph-mon.a.asok config get fsid 2024-09-18T16:13:18.465 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T16:13:18.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:13:18.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:13:18.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T16:13:18.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T16:13:18.466 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:13:18.467 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:13:18.467 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:13:18.467 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:13:18.467 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:13:18.467 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:13:18.468 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.495962/ceph-mon.a.asok 2024-09-18T16:13:18.469 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:13:18.469 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.495962/ceph-mon.a.asok config get mon_host 2024-09-18T16:13:18.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:729: _scrub_snaps_multi: run_mgr td/osd-scrub-snaps x 2024-09-18T16:13:18.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-snaps 2024-09-18T16:13:18.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T16:13:18.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T16:13:18.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T16:13:18.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-snaps/x 2024-09-18T16:13:18.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T16:13:19.424 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T16:13:19.424 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:13:19.424 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:13:19.425 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:13:19.425 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:13:19.425 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:13:19.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.495962/$cluster-$name.asok' 2024-09-18T16:13:19.426 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:13:19.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:13:19.469 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:730: _scrub_snaps_multi: expr 2 - 1 2024-09-18T16:13:19.470 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:730: _scrub_snaps_multi: seq 0 1 2024-09-18T16:13:19.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:730: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:13:19.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:732: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 0 2024-09-18T16:13:19.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-snaps 2024-09-18T16:13:19.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:13:19.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T16:13:19.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:13:19.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-snaps/0 2024-09-18T16:13:19.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3' 2024-09-18T16:13:19.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:13:19.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:13:19.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:13:19.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-09-18T16:13:19.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-09-18T16:13:19.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:13:19.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:13:19.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-18T16:13:19.477 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:13:19.477 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:13:19.477 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:13:19.477 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:13:19.477 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:13:19.477 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:13:19.477 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.495962/$cluster-$name.asok' 2024-09-18T16:13:19.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' 2024-09-18T16:13:19.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:13:19.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:13:19.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:13:19.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-18T16:13:19.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-18T16:13:19.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:13:19.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:13:19.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:13:19.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:13:19.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:13:19.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:13:19.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-snaps/0 2024-09-18T16:13:19.479 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:13:19.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6b191d7a-8623-4e4d-a9ac-51ed8902d35a 2024-09-18T16:13:19.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6b191d7a-8623-4e4d-a9ac-51ed8902d35a' 2024-09-18T16:13:19.479 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 6b191d7a-8623-4e4d-a9ac-51ed8902d35a 2024-09-18T16:13:19.479 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:13:19.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAf/OpmzKyOHRAAPdeBK8/92IdKSAxlOTeIGA== 2024-09-18T16:13:19.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAf/OpmzKyOHRAAPdeBK8/92IdKSAxlOTeIGA=="}' 2024-09-18T16:13:19.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6b191d7a-8623-4e4d-a9ac-51ed8902d35a -i td/osd-scrub-snaps/0/new.json 2024-09-18T16:13:19.781 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:13:19.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-snaps/0/new.json 2024-09-18T16:13:19.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAf/OpmzKyOHRAAPdeBK8/92IdKSAxlOTeIGA== --osd-uuid 6b191d7a-8623-4e4d-a9ac-51ed8902d35a 2024-09-18T16:13:19.831 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:13:19.827+0000 7fd9cc2df540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:13:19.874 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:13:19.871+0000 7fd9cc2df540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:13:19.876 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:13:19.871+0000 7fd9cc2df540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:13:19.877 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:13:19.871+0000 7fd9cc2df540 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-09-18T16:13:19.877 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:13:19.871+0000 7fd9cc2df540 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-09-18T16:13:19.877 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:13:19.871+0000 7fd9cc2df540 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-09-18T16:13:19.921 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:13:19.915+0000 7fd9cc2df540 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2024-09-18T16:13:26.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2024-09-18T16:13:26.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:13:26.029 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T16:13:26.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T16:13:26.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:13:26.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T16:13:26.420 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T16:13:26.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:13:26.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:13:26.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:13:26.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:13:26.453 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:13:26.451+0000 7fa87b501540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:13:26.456 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:13:26.451+0000 7fa87b501540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:13:26.457 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:13:26.451+0000 7fa87b501540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:13:26.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T16:13:26.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:13:26.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:13:26.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:13:26.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:13:26.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:26.785 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:13:26.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:13:26.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:26.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:13:27.000 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:13:26.995+0000 7fa87b501540 -1 Falling back to public interface 2024-09-18T16:13:27.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:28.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:28.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:28.130 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:13:28.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:13:28.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:28.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:13:28.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:29.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:29.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:29.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:13:29.451 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:13:29.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:29.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:13:29.666 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:13:29.663+0000 7fa87b501540 -1 osd.0 0 log_to_monitors true 2024-09-18T16:13:29.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:30.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:30.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:30.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:13:30.772 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:13:30.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:30.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:13:31.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:32.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:32.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:32.091 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:13:32.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:13:32.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:32.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:13:32.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:33.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:33.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:33.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:13:33.564 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:13:33.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:33.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:13:33.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:34.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:34.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:34.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:13:34.880 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:13:34.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:34.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:13:35.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:36.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:36.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:36.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:13:36.206 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:13:36.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:36.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:13:36.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:37.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:37.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:37.584 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:13:37.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:13:37.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:37.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:13:37.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:38.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:38.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:38.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:13:38.901 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:13:38.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:38.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:13:39.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:40.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:40.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:40.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:13:40.214 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:13:40.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:40.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:13:40.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:41.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:41.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:41.629 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:13:41.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:13:41.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:41.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:13:41.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:42.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:42.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:42.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:13:42.949 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:13:42.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:42.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:13:43.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:44.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:44.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:44.385 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:13:44.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:13:44.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:44.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:13:44.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:45.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:45.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:45.843 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:13:45.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:13:45.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:45.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:13:46.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:47.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:47.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:47.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:13:47.628 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:13:47.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:47.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:13:48.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:49.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:49.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:49.057 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:13:49.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:13:49.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:49.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:13:49.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:50.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:50.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:50.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:13:50.643 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:13:50.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:50.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:13:51.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:52.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:52.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:52.433 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:13:52.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:13:52.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:52.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:13:52.801 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/540554,v1:127.0.0.1:6803/540554] [v2:127.0.0.1:6804/540554,v1:127.0.0.1:6805/540554] exists,up 6b191d7a-8623-4e4d-a9ac-51ed8902d35a 2024-09-18T16:13:52.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:13:52.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:13:52.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:13:52.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:730: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:13:52.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:732: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 1 2024-09-18T16:13:52.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-snaps 2024-09-18T16:13:52.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:13:52.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T16:13:52.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:13:52.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-snaps/1 2024-09-18T16:13:52.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3' 2024-09-18T16:13:52.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:13:52.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:13:52.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:13:52.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-09-18T16:13:52.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-09-18T16:13:52.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:13:52.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:13:52.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-18T16:13:52.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:13:52.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:13:52.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:13:52.805 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:13:52.806 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:13:52.806 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:13:52.807 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.495962/$cluster-$name.asok' 2024-09-18T16:13:52.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' 2024-09-18T16:13:52.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:13:52.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:13:52.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:13:52.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-18T16:13:52.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-18T16:13:52.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:13:52.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:13:52.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:13:52.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:13:52.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:13:52.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:13:52.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-snaps/1 2024-09-18T16:13:52.810 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:13:52.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3bb4ac15-2392-41f3-a33a-9124295a1586 2024-09-18T16:13:52.810 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 3bb4ac15-2392-41f3-a33a-9124295a1586 2024-09-18T16:13:52.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 3bb4ac15-2392-41f3-a33a-9124295a1586' 2024-09-18T16:13:52.811 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:13:52.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBA/OpmDSFlMRAAyjctBfZbpvpiG4n+fpsEBg== 2024-09-18T16:13:52.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBA/OpmDSFlMRAAyjctBfZbpvpiG4n+fpsEBg=="}' 2024-09-18T16:13:52.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3bb4ac15-2392-41f3-a33a-9124295a1586 -i td/osd-scrub-snaps/1/new.json 2024-09-18T16:13:53.245 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:13:53.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-snaps/1/new.json 2024-09-18T16:13:53.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBA/OpmDSFlMRAAyjctBfZbpvpiG4n+fpsEBg== --osd-uuid 3bb4ac15-2392-41f3-a33a-9124295a1586 2024-09-18T16:13:53.299 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:13:53.295+0000 7f3c43624540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:13:53.310 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:13:53.307+0000 7f3c43624540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:13:53.311 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:13:53.307+0000 7f3c43624540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:13:53.313 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:13:53.307+0000 7f3c43624540 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2024-09-18T16:13:53.313 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:13:53.307+0000 7f3c43624540 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2024-09-18T16:13:53.313 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:13:53.307+0000 7f3c43624540 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2024-09-18T16:13:53.384 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:13:53.379+0000 7f3c43624540 -1 bluestore(td/osd-scrub-snaps/1) _read_fsid unparsable uuid 2024-09-18T16:13:59.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-snaps/1/keyring 2024-09-18T16:13:59.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:13:59.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T16:13:59.341 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T16:13:59.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-snaps/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:13:59.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T16:13:59.731 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T16:13:59.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:13:59.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:13:59.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:13:59.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:13:59.762 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:13:59.759+0000 7f18fe9c5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:13:59.766 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:13:59.763+0000 7f18fe9c5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:13:59.767 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:13:59.763+0000 7f18fe9c5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:14:00.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T16:14:00.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:14:00.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T16:14:00.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:14:00.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:14:00.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:00.071 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:14:00.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:14:00.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:00.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:14:00.299 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:14:00.295+0000 7f18fe9c5540 -1 Falling back to public interface 2024-09-18T16:14:00.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:01.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:01.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:01.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:14:01.417 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:14:01.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:01.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:14:01.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:02.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:02.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:02.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:14:02.758 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:14:02.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:02.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:14:03.061 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:14:03.059+0000 7f18fe9c5540 -1 osd.1 0 log_to_monitors true 2024-09-18T16:14:03.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:04.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:04.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:04.089 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:14:04.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:14:04.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:04.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:14:04.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:05.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:05.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:05.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:14:05.403 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:14:05.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:05.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:14:05.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:06.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:06.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:06.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:14:06.792 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:14:06.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:06.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:14:07.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:08.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:08.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:08.152 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:14:08.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:14:08.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:08.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:14:08.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:09.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:09.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:09.471 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:14:09.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:14:09.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:09.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:14:09.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:10.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:10.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:10.831 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:14:10.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:14:10.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:10.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:14:11.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:12.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:12.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:12.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:14:12.157 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:14:12.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:12.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:14:12.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:13.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:13.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:13.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:14:13.467 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:14:13.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:13.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:14:13.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:14.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:14.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:14.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:14:14.833 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:14:14.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:14.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:14:15.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:16.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:16.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:16.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:14:16.352 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:14:16.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:16.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:14:16.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:17.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:17.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:17.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:14:17.843 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:14:17.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:17.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:14:18.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:19.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:19.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:19.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:14:19.467 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:14:19.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:19.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:14:20.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:21.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:21.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:21.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:14:21.062 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:14:21.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:21.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:14:21.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:22.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:22.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:22.956 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:14:22.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:14:22.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:22.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:14:23.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:24.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:24.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:24.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:14:24.540 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:14:24.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:24.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:14:25.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:26.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:26.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:26.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:14:26.228 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:14:26.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:14:26.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:26.576 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/542333,v1:127.0.0.1:6811/542333] [v2:127.0.0.1:6812/542333,v1:127.0.0.1:6813/542333] exists,up 3bb4ac15-2392-41f3-a33a-9124295a1586 2024-09-18T16:14:26.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:14:26.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:14:26.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:14:26.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:736: _scrub_snaps_multi: ceph osd set noscrub 2024-09-18T16:14:27.161 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T16:14:27.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:737: _scrub_snaps_multi: ceph osd set nodeep-scrub 2024-09-18T16:14:27.689 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T16:14:27.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: create_pool test 1 1 2024-09-18T16:14:27.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-18T16:14:28.275 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'test' created 2024-09-18T16:14:28.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T16:14:29.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:741: _scrub_snaps_multi: wait_for_clean 2024-09-18T16:14:29.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:14:29.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:14:29.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:14:29.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:14:29.293 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:14:29.293 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:14:29.293 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:14:29.294 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:14:29.294 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:14:29.294 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:14:29.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:14:29.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:14:29.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:14:29.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:14:29.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:14:29.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:14:29.867 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T16:14:29.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:14:29.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:14:29.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:14:30.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836495 2024-09-18T16:14:30.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836495 2024-09-18T16:14:30.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836495' 2024-09-18T16:14:30.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:14:30.041 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:14:30.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672970 2024-09-18T16:14:30.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672970 2024-09-18T16:14:30.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836495 1-42949672970' 2024-09-18T16:14:30.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:14:30.190 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836495 2024-09-18T16:14:30.190 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:14:30.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:14:30.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836495 2024-09-18T16:14:30.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:14:30.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836495 2024-09-18T16:14:30.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-09-18T16:14:30.195 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836495 2024-09-18T16:14:30.196 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:14:30.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836494 -lt 21474836495 2024-09-18T16:14:30.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:14:31.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:14:31.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:14:31.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836495 -lt 21474836495 2024-09-18T16:14:31.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:14:31.847 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672970 2024-09-18T16:14:31.847 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:14:31.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:14:31.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672970 2024-09-18T16:14:31.851 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:14:31.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672970 2024-09-18T16:14:31.852 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672970 2024-09-18T16:14:31.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-09-18T16:14:31.853 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:14:32.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672970 -lt 42949672970 2024-09-18T16:14:32.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:14:32.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:14:32.194 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:14:32.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:14:32.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:14:32.602 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:14:32.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:14:32.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:14:32.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:14:32.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:14:32.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:14:32.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:14:32.950 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:14:32.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:14:32.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:14:33.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:14:33.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:14:33.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:14:33.318 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: ceph osd dump 2024-09-18T16:14:33.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: awk '{ print $2 }' 2024-09-18T16:14:33.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: grep '^pool.*['\'']test['\'']' 2024-09-18T16:14:33.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: poolid=1 2024-09-18T16:14:33.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: dd if=/dev/urandom of=testdata.495962 bs=1032 count=1 2024-09-18T16:14:33.655 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records in 2024-09-18T16:14:33.655 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records out 2024-09-18T16:14:33.655 INFO:tasks.workunit.client.0.smithi046.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 0.000109108 s, 9.5 MB/s 2024-09-18T16:14:33.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: seq 1 16 2024-09-18T16:14:33.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:14:33.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj1 testdata.495962 2024-09-18T16:14:33.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:14:33.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj2 testdata.495962 2024-09-18T16:14:33.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:14:33.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj3 testdata.495962 2024-09-18T16:14:34.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:14:34.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj4 testdata.495962 2024-09-18T16:14:34.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:14:34.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj5 testdata.495962 2024-09-18T16:14:34.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:14:34.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj6 testdata.495962 2024-09-18T16:14:34.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:14:34.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj7 testdata.495962 2024-09-18T16:14:34.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:14:34.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj8 testdata.495962 2024-09-18T16:14:34.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:14:34.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj9 testdata.495962 2024-09-18T16:14:34.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:14:34.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj10 testdata.495962 2024-09-18T16:14:34.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:14:34.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj11 testdata.495962 2024-09-18T16:14:35.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:14:35.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj12 testdata.495962 2024-09-18T16:14:35.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:14:35.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj13 testdata.495962 2024-09-18T16:14:35.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:14:35.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj14 testdata.495962 2024-09-18T16:14:35.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:14:35.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj15 testdata.495962 2024-09-18T16:14:35.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:745: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-18T16:14:35.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: rados -p test put obj16 testdata.495962 2024-09-18T16:14:35.815 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:750: _scrub_snaps_multi: get_primary test obj1 2024-09-18T16:14:35.815 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T16:14:35.815 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T16:14:35.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T16:14:35.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T16:14:36.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:750: _scrub_snaps_multi: local primary=1 2024-09-18T16:14:36.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:751: _scrub_snaps_multi: get_not_primary test obj1 2024-09-18T16:14:36.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-09-18T16:14:36.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-09-18T16:14:36.149 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-09-18T16:14:36.149 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T16:14:36.149 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T16:14:36.149 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T16:14:36.150 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T16:14:36.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T16:14:36.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-09-18T16:14:36.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T16:14:36.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:751: _scrub_snaps_multi: local replica=0 2024-09-18T16:14:36.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:753: _scrub_snaps_multi: eval create_scenario td/osd-scrub-snaps test testdata.495962 '$replica' 2024-09-18T16:14:36.879 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:753: _scrub_snaps_multi: create_scenario td/osd-scrub-snaps test testdata.495962 0 2024-09-18T16:14:36.879 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2024-09-18T16:14:36.879 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2024-09-18T16:14:36.879 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.495962 2024-09-18T16:14:36.879 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=0 2024-09-18T16:14:36.879 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2024-09-18T16:14:36.880 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2024-09-18T16:14:37.081 INFO:tasks.workunit.client.0.smithi046.stdout:created pool test snap snap1 2024-09-18T16:14:37.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.495962 bs=256 count=1 2024-09-18T16:14:37.087 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records in 2024-09-18T16:14:37.087 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records out 2024-09-18T16:14:37.087 INFO:tasks.workunit.client.0.smithi046.stderr:256 bytes copied, 0.000248178 s, 1.0 MB/s 2024-09-18T16:14:37.088 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.495962 2024-09-18T16:14:37.351 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.495962 2024-09-18T16:14:37.468 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.495962 2024-09-18T16:14:37.619 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2024-09-18T16:14:37.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:14:37.621 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.495962 2024-09-18T16:14:37.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:14:37.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.495962 2024-09-18T16:14:37.878 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:14:37.878 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.495962 2024-09-18T16:14:38.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:14:38.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.495962 2024-09-18T16:14:38.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:14:38.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.495962 2024-09-18T16:14:38.272 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:14:38.272 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.495962 2024-09-18T16:14:38.422 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:14:38.422 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.495962 2024-09-18T16:14:38.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:14:38.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.495962 2024-09-18T16:14:38.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-18T16:14:38.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.495962 2024-09-18T16:14:38.849 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2024-09-18T16:14:38.849 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2024-09-18T16:14:39.065 INFO:tasks.workunit.client.0.smithi046.stdout:created pool test snap snap2 2024-09-18T16:14:39.069 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.495962 bs=256 count=2 2024-09-18T16:14:39.071 INFO:tasks.workunit.client.0.smithi046.stderr:2+0 records in 2024-09-18T16:14:39.071 INFO:tasks.workunit.client.0.smithi046.stderr:2+0 records out 2024-09-18T16:14:39.071 INFO:tasks.workunit.client.0.smithi046.stderr:512 bytes copied, 0.000253618 s, 2.0 MB/s 2024-09-18T16:14:39.071 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.495962 2024-09-18T16:14:39.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2024-09-18T16:14:39.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2024-09-18T16:14:39.459 INFO:tasks.workunit.client.0.smithi046.stdout:created pool test snap snap3 2024-09-18T16:14:39.463 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.495962 bs=256 count=3 2024-09-18T16:14:39.465 INFO:tasks.workunit.client.0.smithi046.stderr:3+0 records in 2024-09-18T16:14:39.465 INFO:tasks.workunit.client.0.smithi046.stderr:3+0 records out 2024-09-18T16:14:39.465 INFO:tasks.workunit.client.0.smithi046.stderr:768 bytes copied, 0.000256518 s, 3.0 MB/s 2024-09-18T16:14:39.465 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.495962 2024-09-18T16:14:39.619 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2024-09-18T16:14:39.619 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2024-09-18T16:14:39.794 INFO:tasks.workunit.client.0.smithi046.stdout:created pool test snap snap4 2024-09-18T16:14:39.798 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.495962 bs=256 count=4 2024-09-18T16:14:39.800 INFO:tasks.workunit.client.0.smithi046.stderr:4+0 records in 2024-09-18T16:14:39.800 INFO:tasks.workunit.client.0.smithi046.stderr:4+0 records out 2024-09-18T16:14:39.800 INFO:tasks.workunit.client.0.smithi046.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000288681 s, 3.5 MB/s 2024-09-18T16:14:39.800 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.495962 2024-09-18T16:14:40.046 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.495962 2024-09-18T16:14:40.189 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2024-09-18T16:14:40.189 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2024-09-18T16:14:40.372 INFO:tasks.workunit.client.0.smithi046.stdout:created pool test snap snap5 2024-09-18T16:14:40.376 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2024-09-18T16:14:40.376 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2024-09-18T16:14:40.569 INFO:tasks.workunit.client.0.smithi046.stdout:created pool test snap snap6 2024-09-18T16:14:40.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.495962 bs=256 count=6 2024-09-18T16:14:40.575 INFO:tasks.workunit.client.0.smithi046.stderr:6+0 records in 2024-09-18T16:14:40.575 INFO:tasks.workunit.client.0.smithi046.stderr:6+0 records out 2024-09-18T16:14:40.575 INFO:tasks.workunit.client.0.smithi046.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 0.000275501 s, 5.6 MB/s 2024-09-18T16:14:40.576 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.495962 2024-09-18T16:14:40.791 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2024-09-18T16:14:40.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2024-09-18T16:14:40.983 INFO:tasks.workunit.client.0.smithi046.stdout:created pool test snap snap7 2024-09-18T16:14:40.987 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2024-09-18T16:14:41.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2024-09-18T16:14:41.336 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2024-09-18T16:14:41.470 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2024-09-18T16:14:41.471 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:14:41.471 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:14:41.471 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:14:41.472 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:14:41.472 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:14:41.786 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:14:41.787 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj1 2024-09-18T16:14:43.843 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:14:43.843 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2024-09-18T16:14:44.743 INFO:tasks.workunit.client.0.smithi046.stdout:WARNING: only removing head with clones present 2024-09-18T16:14:44.743 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:ff7b1f36:::obj1:head# 2024-09-18T16:14:45.541 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-09-18T16:14:45.541 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2024-09-18T16:14:46.984 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:14:46.984 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-18T16:14:47.843 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:c52c9666:::obj5:2# 2024-09-18T16:14:48.380 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-09-18T16:14:48.381 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2024-09-18T16:14:49.881 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:14:49.881 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:14:49.881 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-18T16:14:51.194 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-18T16:14:51.196 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:14:51.196 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-18T16:14:51.196 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:14:51.197 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-18T16:14:51.197 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-18T16:14:51.197 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-18T16:14:51.197 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-18T16:14:51.197 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-18T16:14:51.197 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-18T16:14:51.197 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-18T16:14:51.197 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:14:51.197 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-18T16:14:51.197 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-18T16:14:51.197 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-18T16:14:51.198 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-18T16:14:51.198 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:14:51.198 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-18T16:14:51.198 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:14:51.198 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:14:51.198 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:14:51.198 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-18T16:14:51.198 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:14:51.198 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-18T16:14:51.198 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:14:51.198 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:14:51.199 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:14:51.199 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-18T16:14:51.199 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:14:51.199 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:14:51.199 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:14:51.199 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-18T16:14:51.199 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:14:51.199 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:14:51.199 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:14:51.199 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-18T16:14:51.199 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:14:51.199 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:14:51.199 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:14:51.199 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:14:51.200 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-18T16:14:51.200 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-18T16:14:51.200 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-18T16:14:51.984 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:c52c9666:::obj5:1# 2024-09-18T16:14:52.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-18T16:14:53.838 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-18T16:14:53.840 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:14:53.840 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-18T16:14:53.840 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:14:53.840 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-18T16:14:53.840 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-18T16:14:53.840 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-18T16:14:53.840 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-18T16:14:53.841 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-18T16:14:53.841 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-18T16:14:53.841 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-18T16:14:53.841 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:14:53.841 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-18T16:14:53.841 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-18T16:14:53.841 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-18T16:14:53.841 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-18T16:14:53.841 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:14:53.841 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-18T16:14:53.841 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:14:53.841 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:14:53.841 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:14:53.842 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-18T16:14:53.842 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:14:53.842 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-18T16:14:53.842 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:14:53.842 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:14:53.842 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:14:53.842 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-18T16:14:53.842 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:14:53.842 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:14:53.842 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:14:53.842 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-18T16:14:53.843 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:14:53.843 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:14:53.843 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:14:53.843 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-18T16:14:53.843 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:14:53.843 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-09-18T16:14:53.843 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:14:53.843 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:14:53.843 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-18T16:14:53.844 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-18T16:14:53.844 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-09-18T16:14:53.846 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-09-18T16:14:53.846 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2024-09-18T16:14:55.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:14:55.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.495962 bs=256 count=18 2024-09-18T16:14:55.149 INFO:tasks.workunit.client.0.smithi046.stderr:18+0 records in 2024-09-18T16:14:55.149 INFO:tasks.workunit.client.0.smithi046.stderr:18+0 records out 2024-09-18T16:14:55.150 INFO:tasks.workunit.client.0.smithi046.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000409433 s, 11.3 MB/s 2024-09-18T16:14:55.150 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.495962 2024-09-18T16:14:56.592 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj3 2024-09-18T16:14:58.283 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:14:58.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.495962 bs=256 count=15 2024-09-18T16:14:58.285 INFO:tasks.workunit.client.0.smithi046.stderr:15+0 records in 2024-09-18T16:14:58.285 INFO:tasks.workunit.client.0.smithi046.stderr:15+0 records out 2024-09-18T16:14:58.285 INFO:tasks.workunit.client.0.smithi046.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.00035034 s, 11.0 MB/s 2024-09-18T16:14:58.286 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.495962 2024-09-18T16:14:59.677 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj4 2024-09-18T16:14:59.677 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2024-09-18T16:15:01.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:15:01.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-18T16:15:01.909 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:0ee9ae15:::obj4:7# 2024-09-18T16:15:02.443 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-18T16:15:03.758 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-18T16:15:03.760 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-18T16:15:03.760 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:15:03.760 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-18T16:15:03.760 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-18T16:15:03.760 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-18T16:15:03.760 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-18T16:15:03.760 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-18T16:15:03.761 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-18T16:15:03.761 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-18T16:15:03.761 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:15:03.761 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-18T16:15:03.761 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-18T16:15:03.761 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-18T16:15:03.761 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-18T16:15:03.761 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-18T16:15:03.761 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:15:03.761 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:15:03.761 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-18T16:15:03.761 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:15:03.761 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-18T16:15:03.762 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:15:03.762 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:15:03.762 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-18T16:15:03.762 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:15:03.762 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:15:03.762 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-18T16:15:03.762 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:15:03.762 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:15:03.762 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-18T16:15:03.762 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:15:03.762 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:15:03.762 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:15:03.763 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-18T16:15:03.763 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:15:03.763 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:15:03.763 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:15:03.763 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:15:03.763 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:15:03.763 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:15:03.763 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-18T16:15:03.764 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj16 2024-09-18T16:15:03.764 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2024-09-18T16:15:05.092 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:15:05.092 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-18T16:15:05.940 INFO:tasks.workunit.client.0.smithi046.stdout:remove #1:461f8b5e:::obj16:7# 2024-09-18T16:15:06.483 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-18T16:15:08.047 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-18T16:15:08.049 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-18T16:15:08.049 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-18T16:15:08.049 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-18T16:15:08.049 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-18T16:15:08.049 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-18T16:15:08.049 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-18T16:15:08.049 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-18T16:15:08.049 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-18T16:15:08.049 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:15:08.050 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-18T16:15:08.050 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-18T16:15:08.050 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-18T16:15:08.050 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-18T16:15:08.050 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-18T16:15:08.050 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:15:08.050 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-18T16:15:08.050 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-18T16:15:08.050 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:15:08.050 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-18T16:15:08.050 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-18T16:15:08.051 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-18T16:15:08.051 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:15:08.051 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-18T16:15:08.051 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:15:08.051 INFO:tasks.workunit.client.0.smithi046.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-18T16:15:08.051 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-18T16:15:08.051 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-09-18T16:15:08.054 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj2 2024-09-18T16:15:09.159 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:15:09.159 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2024-09-18T16:15:10.552 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:15:10.552 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2024-09-18T16:15:10.556 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:15:10.556 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.495962 bs=256 count=7 2024-09-18T16:15:10.557 INFO:tasks.workunit.client.0.smithi046.stderr:7+0 records in 2024-09-18T16:15:10.557 INFO:tasks.workunit.client.0.smithi046.stderr:7+0 records out 2024-09-18T16:15:10.557 INFO:tasks.workunit.client.0.smithi046.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 0.00021882 s, 8.2 MB/s 2024-09-18T16:15:10.557 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.495962 2024-09-18T16:15:12.360 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj6 2024-09-18T16:15:13.238 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:15:13.775 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:15:13.775 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2024-09-18T16:15:15.076 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj7 2024-09-18T16:15:15.998 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:15:16.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:15:16.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2024-09-18T16:15:17.940 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj8 2024-09-18T16:15:18.797 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:15:19.331 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:15:19.331 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2024-09-18T16:15:20.660 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj9 2024-09-18T16:15:21.518 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:15:22.303 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:15:22.303 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2024-09-18T16:15:23.624 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj10 2024-09-18T16:15:24.482 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:15:25.015 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:15:25.015 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2024-09-18T16:15:26.412 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj11 2024-09-18T16:15:27.319 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:15:27.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:15:27.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2024-09-18T16:15:29.180 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj12 2024-09-18T16:15:30.066 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:15:30.599 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:15:30.599 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2024-09-18T16:15:31.944 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj13 2024-09-18T16:15:32.864 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:15:33.399 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:15:33.399 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2024-09-18T16:15:34.736 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj14 2024-09-18T16:15:35.589 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:15:36.375 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:15:36.375 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2024-09-18T16:15:37.763 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2024-09-18T16:15:37.764 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj15 2024-09-18T16:15:38.654 INFO:tasks.workunit.client.0.smithi046.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-18T16:15:39.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:15:39.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2024-09-18T16:15:40.559 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2024-09-18T16:15:40.560 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2024-09-18T16:15:40.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: rm -f testdata.495962 2024-09-18T16:15:40.564 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: expr 2 - 1 2024-09-18T16:15:40.565 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: seq 0 1 2024-09-18T16:15:40.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:15:40.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 0 2024-09-18T16:15:40.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-snaps 2024-09-18T16:15:40.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T16:15:40.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T16:15:40.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T16:15:40.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-snaps/0 2024-09-18T16:15:40.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3' 2024-09-18T16:15:40.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:15:40.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:15:40.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:15:40.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-09-18T16:15:40.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-09-18T16:15:40.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T16:15:40.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T16:15:40.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-18T16:15:40.570 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T16:15:40.570 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:15:40.570 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:15:40.570 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:15:40.570 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:15:40.570 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:15:40.570 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.495962/$cluster-$name.asok' 2024-09-18T16:15:40.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' 2024-09-18T16:15:40.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:15:40.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-18T16:15:40.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-18T16:15:40.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:15:40.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:15:40.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:15:40.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:15:40.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T16:15:40.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T16:15:40.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-snaps/0 2024-09-18T16:15:40.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T16:15:40.572 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T16:15:40.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:15:40.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-snaps/0/whoami 2024-09-18T16:15:40.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T16:15:40.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T16:15:40.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T16:15:40.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T16:15:40.607 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:15:40.602+0000 7f47e6656540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:15:40.610 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:15:40.606+0000 7f47e6656540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:15:40.613 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:15:40.606+0000 7f47e6656540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:15:40.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T16:15:40.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:15:40.892 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:15:40.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:15:40.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:15:40.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:15:40.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:40.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:15:40.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:40.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:15:41.147 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:15:41.142+0000 7f47e6656540 -1 Falling back to public interface 2024-09-18T16:15:41.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:42.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:42.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:42.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:15:42.237 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:15:42.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:42.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:15:42.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:43.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:43.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:43.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:15:43.545 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:15:43.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:43.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:15:43.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:43.935 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:15:43.930+0000 7f47e6656540 -1 osd.0 23 log_to_monitors true 2024-09-18T16:15:44.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:44.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:44.918 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:15:44.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:15:44.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:44.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:15:45.237 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 24 last_clean_interval [5,23) [v2:127.0.0.1:6802/560406,v1:127.0.0.1:6803/560406] [v2:127.0.0.1:6804/560406,v1:127.0.0.1:6805/560406] exists,up 6b191d7a-8623-4e4d-a9ac-51ed8902d35a 2024-09-18T16:15:45.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:15:45.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:15:45.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:15:45.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:15:45.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 1 2024-09-18T16:15:45.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-snaps 2024-09-18T16:15:45.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T16:15:45.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T16:15:45.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T16:15:45.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-snaps/1 2024-09-18T16:15:45.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3' 2024-09-18T16:15:45.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:15:45.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:15:45.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:15:45.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-09-18T16:15:45.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-09-18T16:15:45.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T16:15:45.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T16:15:45.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-18T16:15:45.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T16:15:45.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:15:45.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:15:45.242 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:15:45.243 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:15:45.243 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:15:45.243 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.495962/$cluster-$name.asok' 2024-09-18T16:15:45.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' 2024-09-18T16:15:45.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:15:45.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-18T16:15:45.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-18T16:15:45.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:15:45.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:15:45.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:15:45.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:15:45.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T16:15:45.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T16:15:45.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-snaps/1 2024-09-18T16:15:45.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T16:15:45.245 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T16:15:45.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.495962/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:15:45.245 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-snaps/1/whoami 2024-09-18T16:15:45.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T16:15:45.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T16:15:45.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T16:15:45.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T16:15:45.272 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:15:45.265+0000 7f551cce9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:15:45.275 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:15:45.269+0000 7f551cce9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:15:45.281 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:15:45.269+0000 7f551cce9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:15:45.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T16:15:45.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:15:45.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T16:15:45.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:15:45.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:15:45.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:45.597 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:15:45.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:15:45.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:45.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:15:45.816 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:15:45.809+0000 7f551cce9540 -1 Falling back to public interface 2024-09-18T16:15:45.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:46.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:46.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:46.942 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:15:46.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:15:46.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:46.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:15:47.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:48.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:48.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:48.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:15:48.263 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:15:48.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:48.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:15:48.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:49.113 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:15:49.109+0000 7f551cce9540 -1 osd.1 23 log_to_monitors true 2024-09-18T16:15:49.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:49.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:49.583 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:15:49.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:15:49.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:49.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:15:49.954 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 29 up_thru 29 down_at 24 last_clean_interval [10,23) [v2:127.0.0.1:6810/560984,v1:127.0.0.1:6811/560984] [v2:127.0.0.1:6812/560984,v1:127.0.0.1:6813/560984] exists,up 3bb4ac15-2392-41f3-a33a-9124295a1586 2024-09-18T16:15:49.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:15:49.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:15:49.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:15:49.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: wait_for_clean 2024-09-18T16:15:49.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:15:49.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:15:49.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:15:49.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:15:49.956 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:15:49.957 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:15:49.957 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:15:49.958 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:15:49.958 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:15:49.958 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:15:50.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:15:50.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:15:50.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:15:50.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:15:50.163 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:15:50.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:15:50.501 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T16:15:50.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:15:50.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:15:50.501 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:15:50.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=111669149699 2024-09-18T16:15:50.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 111669149699 2024-09-18T16:15:50.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-111669149699' 2024-09-18T16:15:50.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:15:50.679 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:15:50.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051586 2024-09-18T16:15:50.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051586 2024-09-18T16:15:50.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-111669149699 1-124554051586' 2024-09-18T16:15:50.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:15:50.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-111669149699 2024-09-18T16:15:50.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:15:50.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:15:50.827 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-111669149699 2024-09-18T16:15:50.827 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:15:50.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=111669149699 2024-09-18T16:15:50.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 111669149699' 2024-09-18T16:15:50.829 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 111669149699 2024-09-18T16:15:50.829 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:15:51.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 111669149699 -lt 111669149699 2024-09-18T16:15:51.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:15:51.162 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-124554051586 2024-09-18T16:15:51.162 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:15:51.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:15:51.166 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-124554051586 2024-09-18T16:15:51.166 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:15:51.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051586 2024-09-18T16:15:51.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2024-09-18T16:15:51.168 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 124554051586 2024-09-18T16:15:51.168 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:15:51.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 124554051586 2024-09-18T16:15:51.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:15:52.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:15:52.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:15:52.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 124554051586 2024-09-18T16:15:52.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:15:53.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-18T16:15:53.837 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:15:54.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051586 -lt 124554051586 2024-09-18T16:15:54.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:15:54.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:15:54.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:15:54.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:15:54.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:15:54.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:15:54.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:15:54.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:15:54.574 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:15:54.574 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:15:54.574 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:15:54.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:15:54.919 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:15:54.919 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:15:54.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:15:55.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:15:55.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:15:55.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:15:55.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:764: _scrub_snaps_multi: local pgid=1.0 2024-09-18T16:15:55.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: pg_scrub 1.0 2024-09-18T16:15:55.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: pg_scrub: local pgid=1.0 2024-09-18T16:15:55.293 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: pg_scrub: get_last_scrub_stamp 1.0 2024-09-18T16:15:55.293 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:15:55.293 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:15:55.293 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:15:55.294 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:15:55.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: pg_scrub: local last_scrub=2024-09-18T16:14:28.082713+0000 2024-09-18T16:15:55.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_scrub: ceph pg scrub 1.0 2024-09-18T16:15:55.928 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 1.0 on osd.1 to scrub 2024-09-18T16:15:55.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_scrub: wait_for_scrub 1.0 2024-09-18T16:14:28.082713+0000 2024-09-18T16:15:55.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-18T16:15:55.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T16:14:28.082713+0000 2024-09-18T16:15:55.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T16:15:55.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T16:15:55.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:15:55.938 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:15:55.938 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:15:55.938 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:15:55.938 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:15:55.938 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:15:56.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:14:28.082713+0000 '>' 2024-09-18T16:14:28.082713+0000 2024-09-18T16:15:56.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:15:57.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:15:57.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:15:57.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:15:57.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:15:57.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:15:57.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:15:57.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:15:57.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:14:28.082713+0000 '>' 2024-09-18T16:14:28.082713+0000 2024-09-18T16:15:57.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:15:58.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:15:58.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:15:58.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:15:58.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:15:58.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:15:58.574 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:15:58.574 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:15:58.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:14:28.082713+0000 '>' 2024-09-18T16:14:28.082713+0000 2024-09-18T16:15:58.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:15:59.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:15:59.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:15:59.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:15:59.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:15:59.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:15:59.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:15:59.902 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:16:00.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:14:28.082713+0000 '>' 2024-09-18T16:14:28.082713+0000 2024-09-18T16:16:00.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:16:01.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:16:01.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 300 )) 2024-09-18T16:16:01.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:16:01.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:16:01.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:16:01.221 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:16:01.221 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:16:01.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:15:58.481142+0000 '>' 2024-09-18T16:14:28.082713+0000 2024-09-18T16:16:01.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T16:16:01.536 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.1.log 2024-09-18T16:16:01.536 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: wc -l 2024-09-18T16:16:01.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: test 16 -gt 3 2024-09-18T16:16:01.544 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:770: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:16:01.544 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:770: _scrub_snaps_multi: wc -l 2024-09-18T16:16:01.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:770: _scrub_snaps_multi: test 16 -gt 3 2024-09-18T16:16:01.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:772: _scrub_snaps_multi: rados list-inconsistent-pg test 2024-09-18T16:16:01.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:774: _scrub_snaps_multi: jq '. | length' td/osd-scrub-snaps/json 2024-09-18T16:16:01.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:774: _scrub_snaps_multi: test 1 = 1 2024-09-18T16:16:01.632 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:776: _scrub_snaps_multi: jq -r '.[0]' td/osd-scrub-snaps/json 2024-09-18T16:16:01.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:776: _scrub_snaps_multi: test 1.0 = 1.0 2024-09-18T16:16:01.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:778: _scrub_snaps_multi: rados list-inconsistent-obj 1.0 --format=json-pretty 2024-09-18T16:16:01.714 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:16:01.714 INFO:tasks.workunit.client.0.smithi046.stdout: "epoch": 29, 2024-09-18T16:16:01.714 INFO:tasks.workunit.client.0.smithi046.stdout: "inconsistents": [ 2024-09-18T16:16:01.714 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.714 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:16:01.715 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj5", 2024-09-18T16:16:01.715 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:16:01.715 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:16:01.715 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:16:01.715 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 5 2024-09-18T16:16:01.715 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.715 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.715 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [ 2024-09-18T16:16:01.715 INFO:tasks.workunit.client.0.smithi046.stdout: "missing" 2024-09-18T16:16:01.715 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.715 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:16:01.715 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:16:01.716 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj5", 2024-09-18T16:16:01.716 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:16:01.716 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": 1, 2024-09-18T16:16:01.716 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 1718170787, 2024-09-18T16:16:01.716 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:16:01.716 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:16:01.716 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:16:01.716 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.716 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "17'19", 2024-09-18T16:16:01.716 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "16'5", 2024-09-18T16:16:01.716 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4241.0:1", 2024-09-18T16:16:01.716 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 5, 2024-09-18T16:16:01.716 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:16:01.717 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:14:34.199919+0000", 2024-09-18T16:16:01.717 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:14:34.201434+0000", 2024-09-18T16:16:01.717 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:16:01.717 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:16:01.717 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:16:01.717 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:16:01.717 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.717 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:16:01.717 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:16:01.717 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0x77b1c822", 2024-09-18T16:16:01.717 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:16:01.717 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:16:01.717 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:16:01.717 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:16:01.718 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:16:01.718 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:16:01.718 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.718 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:16:01.718 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.718 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:16:01.718 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.718 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:16:01.718 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:16:01.718 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:16:01.718 INFO:tasks.workunit.client.0.smithi046.stdout: "missing" 2024-09-18T16:16:01.719 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.719 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.719 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.719 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:16:01.719 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:16:01.719 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.719 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032 2024-09-18T16:16:01.719 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.719 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.719 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.719 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.719 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:16:01.719 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj5", 2024-09-18T16:16:01.719 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:16:01.719 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:16:01.720 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 2, 2024-09-18T16:16:01.720 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 20 2024-09-18T16:16:01.720 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.720 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.720 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [ 2024-09-18T16:16:01.720 INFO:tasks.workunit.client.0.smithi046.stdout: "missing" 2024-09-18T16:16:01.720 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.720 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:16:01.720 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:16:01.720 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj5", 2024-09-18T16:16:01.720 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:16:01.720 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": 2, 2024-09-18T16:16:01.720 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 1718170787, 2024-09-18T16:16:01.721 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:16:01.721 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:16:01.721 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:16:01.721 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.721 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "18'41", 2024-09-18T16:16:01.721 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "17'20", 2024-09-18T16:16:01.721 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4289.0:1", 2024-09-18T16:16:01.721 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 20, 2024-09-18T16:16:01.721 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:16:01.721 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:14:37.387385+0000", 2024-09-18T16:16:01.721 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:14:37.388781+0000", 2024-09-18T16:16:01.721 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:16:01.722 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:16:01.722 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:16:01.722 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:16:01.722 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.722 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:16:01.722 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:16:01.722 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0x485963c8", 2024-09-18T16:16:01.722 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:16:01.722 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:16:01.722 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:16:01.723 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:16:01.723 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:16:01.723 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:16:01.723 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.723 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:16:01.723 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.723 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:16:01.723 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.723 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:16:01.723 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:16:01.724 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:16:01.724 INFO:tasks.workunit.client.0.smithi046.stdout: "missing" 2024-09-18T16:16:01.724 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.724 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.724 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.724 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:16:01.724 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:16:01.724 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.724 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256 2024-09-18T16:16:01.724 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.725 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.725 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.725 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.725 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:16:01.725 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj5", 2024-09-18T16:16:01.725 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:16:01.725 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:16:01.725 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 4, 2024-09-18T16:16:01.725 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 42 2024-09-18T16:16:01.725 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.725 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:16:01.725 INFO:tasks.workunit.client.0.smithi046.stdout: "size_mismatch" 2024-09-18T16:16:01.725 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.726 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [ 2024-09-18T16:16:01.726 INFO:tasks.workunit.client.0.smithi046.stdout: "size_mismatch_info", 2024-09-18T16:16:01.726 INFO:tasks.workunit.client.0.smithi046.stdout: "obj_size_info_mismatch" 2024-09-18T16:16:01.726 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.726 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:16:01.726 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:16:01.726 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj5", 2024-09-18T16:16:01.726 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:16:01.726 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": 4, 2024-09-18T16:16:01.726 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 1718170787, 2024-09-18T16:16:01.726 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:16:01.726 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:16:01.727 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:16:01.727 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.727 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "20'45", 2024-09-18T16:16:01.727 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "18'42", 2024-09-18T16:16:01.727 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4325.0:1", 2024-09-18T16:16:01.727 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 42, 2024-09-18T16:16:01.727 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 512, 2024-09-18T16:16:01.727 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:14:39.109297+0000", 2024-09-18T16:16:01.727 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:14:39.119106+0000", 2024-09-18T16:16:01.727 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:16:01.727 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:16:01.727 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:16:01.728 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:16:01.728 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.728 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:16:01.728 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:16:01.728 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0xa4f17110", 2024-09-18T16:16:01.728 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:16:01.728 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:16:01.728 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:16:01.728 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:16:01.728 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:16:01.728 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:16:01.728 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.728 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:16:01.729 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.729 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:16:01.729 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.729 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:16:01.729 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:16:01.729 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:16:01.729 INFO:tasks.workunit.client.0.smithi046.stdout: "size_mismatch_info", 2024-09-18T16:16:01.729 INFO:tasks.workunit.client.0.smithi046.stdout: "obj_size_info_mismatch" 2024-09-18T16:16:01.729 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.729 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 4608, 2024-09-18T16:16:01.729 INFO:tasks.workunit.client.0.smithi046.stdout: "object_info": { 2024-09-18T16:16:01.729 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:16:01.729 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj5", 2024-09-18T16:16:01.730 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:16:01.730 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": 4, 2024-09-18T16:16:01.730 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 1718170787, 2024-09-18T16:16:01.730 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:16:01.730 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:16:01.730 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:16:01.730 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.730 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "20'45", 2024-09-18T16:16:01.730 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "18'42", 2024-09-18T16:16:01.730 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4325.0:1", 2024-09-18T16:16:01.730 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 42, 2024-09-18T16:16:01.730 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 512, 2024-09-18T16:16:01.730 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:14:39.109297+0000", 2024-09-18T16:16:01.730 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:14:39.119106+0000", 2024-09-18T16:16:01.731 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:16:01.731 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:16:01.731 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:16:01.731 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:16:01.731 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.731 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:16:01.731 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:16:01.731 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0xa4f17110", 2024-09-18T16:16:01.731 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:16:01.731 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:16:01.731 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:16:01.731 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:16:01.731 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:16:01.732 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:16:01.732 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.732 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:16:01.732 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.732 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.732 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.732 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:16:01.732 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:16:01.732 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.732 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 512 2024-09-18T16:16:01.732 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.732 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.732 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.732 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.733 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:16:01.733 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj4", 2024-09-18T16:16:01.733 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:16:01.733 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:16:01.733 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 7, 2024-09-18T16:16:01.733 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 4 2024-09-18T16:16:01.733 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.733 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.733 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [ 2024-09-18T16:16:01.733 INFO:tasks.workunit.client.0.smithi046.stdout: "missing" 2024-09-18T16:16:01.733 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.733 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:16:01.733 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:16:01.734 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj4", 2024-09-18T16:16:01.734 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:16:01.734 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": 7, 2024-09-18T16:16:01.734 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 2826278768, 2024-09-18T16:16:01.734 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:16:01.734 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:16:01.734 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:16:01.734 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.734 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "23'51", 2024-09-18T16:16:01.734 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "16'4", 2024-09-18T16:16:01.734 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4238.0:1", 2024-09-18T16:16:01.735 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 4, 2024-09-18T16:16:01.735 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:16:01.735 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:14:34.052851+0000", 2024-09-18T16:16:01.735 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:14:34.054013+0000", 2024-09-18T16:16:01.735 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:16:01.735 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:16:01.735 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:16:01.735 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:16:01.735 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.735 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:16:01.735 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:16:01.735 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0x77b1c822", 2024-09-18T16:16:01.735 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:16:01.736 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:16:01.736 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:16:01.736 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:16:01.736 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:16:01.736 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:16:01.736 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.736 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:16:01.736 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.736 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:16:01.736 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.736 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:16:01.736 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:16:01.736 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:16:01.736 INFO:tasks.workunit.client.0.smithi046.stdout: "missing" 2024-09-18T16:16:01.737 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.737 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.737 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.737 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:16:01.737 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:16:01.737 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.737 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032 2024-09-18T16:16:01.737 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.737 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.737 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.737 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.737 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:16:01.738 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj5", 2024-09-18T16:16:01.738 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:16:01.738 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:16:01.738 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 7, 2024-09-18T16:16:01.738 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 0 2024-09-18T16:16:01.738 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.738 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.738 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [ 2024-09-18T16:16:01.738 INFO:tasks.workunit.client.0.smithi046.stdout: "missing", 2024-09-18T16:16:01.738 INFO:tasks.workunit.client.0.smithi046.stdout: "info_missing" 2024-09-18T16:16:01.738 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.738 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:16:01.739 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.739 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:16:01.739 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:16:01.739 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:16:01.739 INFO:tasks.workunit.client.0.smithi046.stdout: "info_missing" 2024-09-18T16:16:01.739 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.739 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1792 2024-09-18T16:16:01.739 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.739 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.739 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:16:01.739 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:16:01.739 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:16:01.739 INFO:tasks.workunit.client.0.smithi046.stdout: "missing" 2024-09-18T16:16:01.740 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.740 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.740 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.740 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.740 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.740 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:16:01.740 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj1", 2024-09-18T16:16:01.740 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:16:01.740 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:16:01.740 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": "head", 2024-09-18T16:16:01.741 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 18 2024-09-18T16:16:01.741 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.741 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.741 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [ 2024-09-18T16:16:01.741 INFO:tasks.workunit.client.0.smithi046.stdout: "missing" 2024-09-18T16:16:01.741 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.741 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:16:01.741 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:16:01.741 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj1", 2024-09-18T16:16:01.741 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:16:01.741 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": -2, 2024-09-18T16:16:01.742 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 1828249343, 2024-09-18T16:16:01.742 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:16:01.742 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:16:01.742 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:16:01.742 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.742 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "17'18", 2024-09-18T16:16:01.742 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "16'1", 2024-09-18T16:16:01.742 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4286.0:1", 2024-09-18T16:16:01.742 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 18, 2024-09-18T16:16:01.742 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:16:01.742 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:14:37.128610+0000", 2024-09-18T16:16:01.742 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:14:37.244553+0000", 2024-09-18T16:16:01.742 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:16:01.743 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:16:01.743 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:16:01.743 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:16:01.743 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.743 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:16:01.743 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:16:01.743 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0x485963c8", 2024-09-18T16:16:01.743 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:16:01.743 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:16:01.743 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:16:01.743 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:16:01.743 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:16:01.743 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:16:01.743 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.744 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:16:01.744 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.744 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:16:01.744 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.744 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:16:01.744 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:16:01.744 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:16:01.744 INFO:tasks.workunit.client.0.smithi046.stdout: "missing" 2024-09-18T16:16:01.744 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.744 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.744 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.744 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:16:01.745 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:16:01.745 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.745 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256 2024-09-18T16:16:01.745 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.745 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.745 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.745 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.745 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:16:01.745 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj10", 2024-09-18T16:16:01.745 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:16:01.745 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:16:01.745 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": "head", 2024-09-18T16:16:01.745 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 32 2024-09-18T16:16:01.745 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.746 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:16:01.746 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset_inconsistency" 2024-09-18T16:16:01.746 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.746 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [], 2024-09-18T16:16:01.746 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:16:01.746 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:16:01.746 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj10", 2024-09-18T16:16:01.746 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:16:01.746 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": -2, 2024-09-18T16:16:01.746 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 718195851, 2024-09-18T16:16:01.746 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:16:01.746 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:16:01.746 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:16:01.746 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.747 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "17'32", 2024-09-18T16:16:01.747 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "16'10", 2024-09-18T16:16:01.747 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4307.0:1", 2024-09-18T16:16:01.747 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 32, 2024-09-18T16:16:01.747 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:16:01.747 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:14:38.191655+0000", 2024-09-18T16:16:01.747 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:14:38.193131+0000", 2024-09-18T16:16:01.747 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:16:01.747 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:16:01.747 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:16:01.747 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:16:01.747 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.747 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:16:01.748 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:16:01.748 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0x485963c8", 2024-09-18T16:16:01.748 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:16:01.748 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:16:01.748 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:16:01.748 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:16:01.748 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:16:01.748 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:16:01.748 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.748 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:16:01.748 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.748 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:16:01.748 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.748 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:16:01.749 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:16:01.749 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.749 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:16:01.749 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:16:01.749 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:16:01.749 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:16:01.749 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.749 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:16:01.749 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:16:01.749 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "????", 2024-09-18T16:16:01.749 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:16:01.749 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T16:16:01.749 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.749 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.749 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.750 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.750 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.750 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.750 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:16:01.750 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:16:01.750 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.750 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:16:01.750 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:16:01.750 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:16:01.750 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:16:01.750 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.750 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:16:01.750 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:16:01.750 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]", 2024-09-18T16:16:01.751 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:16:01.751 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T16:16:01.751 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.751 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.751 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.751 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.751 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.751 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.751 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.752 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.752 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:16:01.752 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj11", 2024-09-18T16:16:01.752 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:16:01.752 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:16:01.752 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": "head", 2024-09-18T16:16:01.752 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 34 2024-09-18T16:16:01.752 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.752 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:16:01.752 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset_inconsistency" 2024-09-18T16:16:01.752 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.752 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [], 2024-09-18T16:16:01.753 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:16:01.753 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:16:01.753 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj11", 2024-09-18T16:16:01.753 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:16:01.753 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": -2, 2024-09-18T16:16:01.753 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 693400951, 2024-09-18T16:16:01.753 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:16:01.753 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:16:01.753 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:16:01.753 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.753 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "17'34", 2024-09-18T16:16:01.753 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "16'11", 2024-09-18T16:16:01.754 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4310.0:1", 2024-09-18T16:16:01.754 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 34, 2024-09-18T16:16:01.754 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:16:01.754 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:14:38.310502+0000", 2024-09-18T16:16:01.754 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:14:38.311663+0000", 2024-09-18T16:16:01.754 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:16:01.754 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:16:01.754 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:16:01.754 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:16:01.754 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.754 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:16:01.754 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:16:01.755 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0x485963c8", 2024-09-18T16:16:01.755 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:16:01.755 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:16:01.755 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:16:01.755 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:16:01.755 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:16:01.755 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:16:01.755 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.755 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:16:01.755 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.755 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:16:01.755 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.756 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:16:01.756 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:16:01.756 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.756 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:16:01.756 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:16:01.756 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:16:01.756 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [] 2024-09-18T16:16:01.756 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.757 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.757 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.757 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:16:01.757 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:16:01.757 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.757 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:16:01.757 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:16:01.757 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:16:01.757 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:16:01.757 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.757 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:16:01.758 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:16:01.758 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]", 2024-09-18T16:16:01.758 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:16:01.758 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T16:16:01.758 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.758 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.758 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.758 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.758 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.758 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.758 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.758 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.758 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:16:01.759 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj13", 2024-09-18T16:16:01.759 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:16:01.759 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:16:01.759 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": "head", 2024-09-18T16:16:01.759 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 38 2024-09-18T16:16:01.759 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.759 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:16:01.759 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset_inconsistency" 2024-09-18T16:16:01.759 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.759 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [], 2024-09-18T16:16:01.759 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:16:01.759 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:16:01.759 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj13", 2024-09-18T16:16:01.759 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:16:01.760 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": -2, 2024-09-18T16:16:01.760 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 2087409765, 2024-09-18T16:16:01.760 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:16:01.760 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:16:01.760 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:16:01.760 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.760 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "17'38", 2024-09-18T16:16:01.760 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "16'13", 2024-09-18T16:16:01.760 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4316.0:1", 2024-09-18T16:16:01.760 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 38, 2024-09-18T16:16:01.761 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:16:01.761 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:14:38.613603+0000", 2024-09-18T16:16:01.761 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:14:38.614645+0000", 2024-09-18T16:16:01.761 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:16:01.761 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:16:01.761 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:16:01.761 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:16:01.761 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.761 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:16:01.761 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:16:01.761 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0x485963c8", 2024-09-18T16:16:01.761 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:16:01.762 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:16:01.762 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:16:01.762 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:16:01.762 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:16:01.762 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:16:01.762 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.762 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:16:01.762 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.762 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:16:01.762 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.762 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:16:01.762 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:16:01.762 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.763 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:16:01.763 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:16:01.763 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:16:01.763 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:16:01.763 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.763 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:16:01.763 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:16:01.763 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]" 2024-09-18T16:16:01.763 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.763 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.763 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.763 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.764 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.764 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:16:01.764 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:16:01.764 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.764 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:16:01.764 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:16:01.764 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:16:01.764 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:16:01.764 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.764 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:16:01.765 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:16:01.765 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]", 2024-09-18T16:16:01.765 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:16:01.765 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T16:16:01.765 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.765 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.765 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.765 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.765 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.765 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.765 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.766 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.766 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:16:01.766 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj14", 2024-09-18T16:16:01.766 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:16:01.766 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:16:01.766 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": "head", 2024-09-18T16:16:01.766 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 40 2024-09-18T16:16:01.766 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.766 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:16:01.768 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset_inconsistency" 2024-09-18T16:16:01.768 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.768 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [], 2024-09-18T16:16:01.768 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:16:01.768 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:16:01.768 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj14", 2024-09-18T16:16:01.768 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:16:01.769 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": -2, 2024-09-18T16:16:01.769 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 2484217095, 2024-09-18T16:16:01.769 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:16:01.769 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:16:01.769 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:16:01.769 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.769 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "17'40", 2024-09-18T16:16:01.769 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "16'14", 2024-09-18T16:16:01.769 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4319.0:1", 2024-09-18T16:16:01.770 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 40, 2024-09-18T16:16:01.770 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:16:01.770 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:14:38.743756+0000", 2024-09-18T16:16:01.770 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:14:38.745202+0000", 2024-09-18T16:16:01.770 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:16:01.770 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:16:01.770 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:16:01.770 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:16:01.770 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.770 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:16:01.770 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:16:01.771 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0x485963c8", 2024-09-18T16:16:01.771 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:16:01.771 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:16:01.771 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:16:01.771 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:16:01.771 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:16:01.771 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:16:01.771 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.771 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:16:01.771 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.771 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:16:01.771 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.771 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:16:01.772 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:16:01.772 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.772 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:16:01.772 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:16:01.772 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:16:01.772 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:16:01.772 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.772 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:16:01.772 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1033, 2024-09-18T16:16:01.772 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]", 2024-09-18T16:16:01.773 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:16:01.773 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T16:16:01.773 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.773 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.773 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.773 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.773 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.773 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.773 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:16:01.773 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:16:01.773 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.773 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:16:01.773 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:16:01.774 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:16:01.774 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:16:01.774 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.774 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:16:01.774 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:16:01.774 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]", 2024-09-18T16:16:01.774 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:16:01.774 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T16:16:01.774 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.774 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.774 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.774 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.774 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.774 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.775 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.775 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.775 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:16:01.775 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj15", 2024-09-18T16:16:01.775 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:16:01.775 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:16:01.775 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": "head", 2024-09-18T16:16:01.775 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 15 2024-09-18T16:16:01.775 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.775 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.775 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [ 2024-09-18T16:16:01.775 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset_corrupted" 2024-09-18T16:16:01.776 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.776 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:16:01.776 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:16:01.776 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj15", 2024-09-18T16:16:01.776 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:16:01.776 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": -2, 2024-09-18T16:16:01.776 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 612772309, 2024-09-18T16:16:01.776 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:16:01.776 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:16:01.776 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:16:01.776 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.776 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "16'15", 2024-09-18T16:16:01.776 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "0'0", 2024-09-18T16:16:01.777 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4271.0:1", 2024-09-18T16:16:01.777 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 15, 2024-09-18T16:16:01.777 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:16:01.777 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:14:35.534129+0000", 2024-09-18T16:16:01.777 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:14:35.535431+0000", 2024-09-18T16:16:01.777 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:16:01.777 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:16:01.777 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:16:01.777 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:16:01.777 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.778 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:16:01.778 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:16:01.778 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0x77b1c822", 2024-09-18T16:16:01.778 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:16:01.778 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:16:01.778 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:16:01.778 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:16:01.778 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:16:01.778 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:16:01.778 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.778 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:16:01.778 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.778 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:16:01.778 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.779 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:16:01.779 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:16:01.779 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:16:01.779 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset_corrupted" 2024-09-18T16:16:01.779 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.779 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:16:01.779 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": "Z2FyYmFnZQo=" 2024-09-18T16:16:01.779 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.779 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.779 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:16:01.779 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:16:01.779 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.779 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:16:01.779 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:16:01.780 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 0, 2024-09-18T16:16:01.780 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [] 2024-09-18T16:16:01.780 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.780 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.780 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.780 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.780 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.780 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:16:01.781 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj2", 2024-09-18T16:16:01.781 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:16:01.781 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:16:01.781 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": "head", 2024-09-18T16:16:01.781 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 56 2024-09-18T16:16:01.781 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.781 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.781 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [ 2024-09-18T16:16:01.782 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset_missing" 2024-09-18T16:16:01.782 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.782 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:16:01.782 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:16:01.782 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj2", 2024-09-18T16:16:01.782 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:16:01.782 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": -2, 2024-09-18T16:16:01.782 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 1058988552, 2024-09-18T16:16:01.782 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:16:01.783 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:16:01.783 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:16:01.783 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.783 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "23'56", 2024-09-18T16:16:01.783 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "20'48", 2024-09-18T16:16:01.783 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4361.0:1", 2024-09-18T16:16:01.783 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 56, 2024-09-18T16:16:01.783 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 0, 2024-09-18T16:16:01.783 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:14:41.373808+0000", 2024-09-18T16:16:01.783 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:14:41.375063+0000", 2024-09-18T16:16:01.784 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:16:01.784 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:16:01.784 INFO:tasks.workunit.client.0.smithi046.stdout: "whiteout", 2024-09-18T16:16:01.784 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty" 2024-09-18T16:16:01.784 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.784 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:16:01.784 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:16:01.784 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0xffffffff", 2024-09-18T16:16:01.784 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:16:01.784 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:16:01.784 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:16:01.784 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:16:01.784 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:16:01.785 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:16:01.785 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.785 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:16:01.785 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.785 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:16:01.785 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.785 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:16:01.785 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:16:01.785 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:16:01.785 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset_missing" 2024-09-18T16:16:01.785 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.785 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 0 2024-09-18T16:16:01.785 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.785 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.786 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:16:01.786 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:16:01.786 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.786 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 0, 2024-09-18T16:16:01.786 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:16:01.786 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 7, 2024-09-18T16:16:01.786 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:16:01.786 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.786 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 4, 2024-09-18T16:16:01.786 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:16:01.786 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]", 2024-09-18T16:16:01.786 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:16:01.786 INFO:tasks.workunit.client.0.smithi046.stdout: 4, 2024-09-18T16:16:01.787 INFO:tasks.workunit.client.0.smithi046.stdout: 3, 2024-09-18T16:16:01.787 INFO:tasks.workunit.client.0.smithi046.stdout: 2, 2024-09-18T16:16:01.787 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T16:16:01.787 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.787 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.787 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.787 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 7, 2024-09-18T16:16:01.787 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1024, 2024-09-18T16:16:01.787 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]", 2024-09-18T16:16:01.787 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:16:01.788 INFO:tasks.workunit.client.0.smithi046.stdout: 7, 2024-09-18T16:16:01.788 INFO:tasks.workunit.client.0.smithi046.stdout: 6, 2024-09-18T16:16:01.788 INFO:tasks.workunit.client.0.smithi046.stdout: 5 2024-09-18T16:16:01.788 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.788 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.788 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.788 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.788 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.788 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.788 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.788 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.788 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:16:01.789 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj3", 2024-09-18T16:16:01.789 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:16:01.789 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:16:01.789 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": "head", 2024-09-18T16:16:01.789 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 44 2024-09-18T16:16:01.789 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.789 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:16:01.789 INFO:tasks.workunit.client.0.smithi046.stdout: "size_mismatch" 2024-09-18T16:16:01.789 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.789 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [ 2024-09-18T16:16:01.789 INFO:tasks.workunit.client.0.smithi046.stdout: "size_mismatch_info", 2024-09-18T16:16:01.789 INFO:tasks.workunit.client.0.smithi046.stdout: "obj_size_info_mismatch" 2024-09-18T16:16:01.789 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.789 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:16:01.790 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:16:01.790 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj3", 2024-09-18T16:16:01.790 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:16:01.790 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": -2, 2024-09-18T16:16:01.790 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 1643547569, 2024-09-18T16:16:01.790 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:16:01.790 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:16:01.790 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:16:01.790 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.790 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "19'44", 2024-09-18T16:16:01.790 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "17'22", 2024-09-18T16:16:01.790 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4331.0:1", 2024-09-18T16:16:01.790 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 44, 2024-09-18T16:16:01.790 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 768, 2024-09-18T16:16:01.791 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:14:39.502492+0000", 2024-09-18T16:16:01.791 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:14:39.512480+0000", 2024-09-18T16:16:01.791 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:16:01.791 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:16:01.791 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:16:01.791 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:16:01.791 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.791 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:16:01.791 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:16:01.791 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0x69634af8", 2024-09-18T16:16:01.791 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:16:01.791 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:16:01.791 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:16:01.792 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:16:01.792 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:16:01.792 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:16:01.792 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.792 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:16:01.792 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.792 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:16:01.792 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.792 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:16:01.792 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:16:01.792 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:16:01.792 INFO:tasks.workunit.client.0.smithi046.stdout: "size_mismatch_info", 2024-09-18T16:16:01.792 INFO:tasks.workunit.client.0.smithi046.stdout: "obj_size_info_mismatch" 2024-09-18T16:16:01.792 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.793 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 3840, 2024-09-18T16:16:01.793 INFO:tasks.workunit.client.0.smithi046.stdout: "object_info": { 2024-09-18T16:16:01.793 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:16:01.793 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj3", 2024-09-18T16:16:01.793 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:16:01.793 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": -2, 2024-09-18T16:16:01.793 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 1643547569, 2024-09-18T16:16:01.793 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:16:01.793 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:16:01.793 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:16:01.793 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.793 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "19'44", 2024-09-18T16:16:01.793 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "17'22", 2024-09-18T16:16:01.793 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4331.0:1", 2024-09-18T16:16:01.794 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 44, 2024-09-18T16:16:01.794 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 768, 2024-09-18T16:16:01.794 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:14:39.502492+0000", 2024-09-18T16:16:01.794 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:14:39.512480+0000", 2024-09-18T16:16:01.794 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:16:01.794 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:16:01.794 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:16:01.794 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:16:01.794 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.794 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:16:01.794 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:16:01.794 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0x69634af8", 2024-09-18T16:16:01.794 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:16:01.794 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:16:01.795 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:16:01.795 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:16:01.795 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:16:01.795 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:16:01.795 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.795 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:16:01.795 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.795 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.795 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.795 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:16:01.795 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:16:01.795 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.795 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 768 2024-09-18T16:16:01.795 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.796 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.796 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.796 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.796 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:16:01.796 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj6", 2024-09-18T16:16:01.796 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:16:01.796 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:16:01.796 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": "head", 2024-09-18T16:16:01.796 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 24 2024-09-18T16:16:01.796 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.796 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:16:01.796 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset_inconsistency" 2024-09-18T16:16:01.796 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.796 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [], 2024-09-18T16:16:01.797 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:16:01.797 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:16:01.797 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj6", 2024-09-18T16:16:01.797 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:16:01.797 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": -2, 2024-09-18T16:16:01.797 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 2202164420, 2024-09-18T16:16:01.797 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:16:01.797 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:16:01.797 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:16:01.797 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.797 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "17'24", 2024-09-18T16:16:01.797 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "16'6", 2024-09-18T16:16:01.797 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4295.0:1", 2024-09-18T16:16:01.797 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 24, 2024-09-18T16:16:01.797 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:16:01.798 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:14:37.686146+0000", 2024-09-18T16:16:01.798 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:14:37.687518+0000", 2024-09-18T16:16:01.798 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:16:01.798 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:16:01.798 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:16:01.798 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:16:01.798 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.798 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:16:01.798 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:16:01.798 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0x485963c8", 2024-09-18T16:16:01.798 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:16:01.798 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:16:01.799 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:16:01.799 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:16:01.799 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:16:01.799 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:16:01.799 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.799 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:16:01.799 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.799 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:16:01.799 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.799 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:16:01.799 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:16:01.799 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.800 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:16:01.800 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:16:01.800 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:16:01.800 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [] 2024-09-18T16:16:01.800 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.800 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.800 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.800 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:16:01.800 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:16:01.800 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.800 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:16:01.800 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:16:01.801 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:16:01.801 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:16:01.801 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.801 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:16:01.801 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:16:01.801 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]", 2024-09-18T16:16:01.801 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:16:01.801 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T16:16:01.801 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.801 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.801 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.801 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.801 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.802 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.802 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.802 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.802 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:16:01.802 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj7", 2024-09-18T16:16:01.802 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:16:01.802 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:16:01.802 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": "head", 2024-09-18T16:16:01.802 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 26 2024-09-18T16:16:01.802 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.802 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:16:01.802 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset_inconsistency" 2024-09-18T16:16:01.802 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.802 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [], 2024-09-18T16:16:01.802 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:16:01.803 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:16:01.803 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj7", 2024-09-18T16:16:01.803 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:16:01.803 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": -2, 2024-09-18T16:16:01.803 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 1552453721, 2024-09-18T16:16:01.803 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:16:01.803 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:16:01.803 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:16:01.803 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.803 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "17'26", 2024-09-18T16:16:01.803 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "16'7", 2024-09-18T16:16:01.803 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4298.0:1", 2024-09-18T16:16:01.803 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 26, 2024-09-18T16:16:01.803 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:16:01.804 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:14:37.801577+0000", 2024-09-18T16:16:01.804 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:14:37.803190+0000", 2024-09-18T16:16:01.804 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:16:01.804 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:16:01.804 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:16:01.804 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:16:01.804 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.804 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:16:01.804 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:16:01.804 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0x485963c8", 2024-09-18T16:16:01.804 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:16:01.804 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:16:01.804 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:16:01.804 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:16:01.805 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:16:01.805 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:16:01.805 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.805 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:16:01.805 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.805 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:16:01.805 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.805 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:16:01.805 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:16:01.805 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.805 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:16:01.806 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:16:01.806 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 0, 2024-09-18T16:16:01.806 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [] 2024-09-18T16:16:01.806 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.806 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.806 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.806 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:16:01.806 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:16:01.806 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.806 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:16:01.806 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:16:01.806 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:16:01.806 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:16:01.806 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.807 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:16:01.807 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:16:01.807 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]", 2024-09-18T16:16:01.807 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:16:01.807 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T16:16:01.807 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.807 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.807 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.807 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.807 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.807 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.807 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.808 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.808 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:16:01.808 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj8", 2024-09-18T16:16:01.808 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:16:01.808 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:16:01.808 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": "head", 2024-09-18T16:16:01.808 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 28 2024-09-18T16:16:01.808 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.808 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:16:01.808 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset_inconsistency" 2024-09-18T16:16:01.808 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.808 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [], 2024-09-18T16:16:01.808 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:16:01.809 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:16:01.809 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj8", 2024-09-18T16:16:01.809 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:16:01.809 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": -2, 2024-09-18T16:16:01.809 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 2381834917, 2024-09-18T16:16:01.809 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:16:01.809 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:16:01.809 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:16:01.809 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.809 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "17'28", 2024-09-18T16:16:01.809 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "16'8", 2024-09-18T16:16:01.809 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4301.0:1", 2024-09-18T16:16:01.810 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 28, 2024-09-18T16:16:01.810 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:16:01.810 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:14:37.917382+0000", 2024-09-18T16:16:01.810 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:14:37.918727+0000", 2024-09-18T16:16:01.810 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:16:01.810 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:16:01.810 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:16:01.810 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:16:01.810 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.810 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:16:01.810 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:16:01.810 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0x485963c8", 2024-09-18T16:16:01.810 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:16:01.810 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:16:01.811 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:16:01.811 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:16:01.811 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:16:01.811 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:16:01.811 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.811 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:16:01.811 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.811 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:16:01.811 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.811 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:16:01.811 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:16:01.811 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.812 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:16:01.812 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:16:01.812 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 0, 2024-09-18T16:16:01.812 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:16:01.812 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.812 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:16:01.812 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:16:01.812 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]", 2024-09-18T16:16:01.812 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:16:01.812 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T16:16:01.812 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.812 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.812 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.813 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.813 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.813 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.813 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:16:01.813 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:16:01.813 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.813 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:16:01.813 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:16:01.813 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:16:01.813 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:16:01.813 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.813 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:16:01.813 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:16:01.813 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]", 2024-09-18T16:16:01.814 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:16:01.814 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T16:16:01.814 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.814 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.814 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.814 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.814 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.814 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.814 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.814 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.814 INFO:tasks.workunit.client.0.smithi046.stdout: "object": { 2024-09-18T16:16:01.814 INFO:tasks.workunit.client.0.smithi046.stdout: "name": "obj9", 2024-09-18T16:16:01.814 INFO:tasks.workunit.client.0.smithi046.stdout: "nspace": "", 2024-09-18T16:16:01.815 INFO:tasks.workunit.client.0.smithi046.stdout: "locator": "", 2024-09-18T16:16:01.815 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": "head", 2024-09-18T16:16:01.815 INFO:tasks.workunit.client.0.smithi046.stdout: "version": 30 2024-09-18T16:16:01.815 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.815 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [ 2024-09-18T16:16:01.815 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset_inconsistency" 2024-09-18T16:16:01.815 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.815 INFO:tasks.workunit.client.0.smithi046.stdout: "union_shard_errors": [], 2024-09-18T16:16:01.815 INFO:tasks.workunit.client.0.smithi046.stdout: "selected_object_info": { 2024-09-18T16:16:01.815 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": { 2024-09-18T16:16:01.815 INFO:tasks.workunit.client.0.smithi046.stdout: "oid": "obj9", 2024-09-18T16:16:01.815 INFO:tasks.workunit.client.0.smithi046.stdout: "key": "", 2024-09-18T16:16:01.816 INFO:tasks.workunit.client.0.smithi046.stdout: "snapid": -2, 2024-09-18T16:16:01.816 INFO:tasks.workunit.client.0.smithi046.stdout: "hash": 3833113727, 2024-09-18T16:16:01.816 INFO:tasks.workunit.client.0.smithi046.stdout: "max": 0, 2024-09-18T16:16:01.816 INFO:tasks.workunit.client.0.smithi046.stdout: "pool": 1, 2024-09-18T16:16:01.816 INFO:tasks.workunit.client.0.smithi046.stdout: "namespace": "" 2024-09-18T16:16:01.816 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.816 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "17'30", 2024-09-18T16:16:01.816 INFO:tasks.workunit.client.0.smithi046.stdout: "prior_version": "16'9", 2024-09-18T16:16:01.816 INFO:tasks.workunit.client.0.smithi046.stdout: "last_reqid": "client.4304.0:1", 2024-09-18T16:16:01.816 INFO:tasks.workunit.client.0.smithi046.stdout: "user_version": 30, 2024-09-18T16:16:01.816 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:16:01.816 INFO:tasks.workunit.client.0.smithi046.stdout: "mtime": "2024-09-18T16:14:38.067594+0000", 2024-09-18T16:16:01.817 INFO:tasks.workunit.client.0.smithi046.stdout: "local_mtime": "2024-09-18T16:14:38.069005+0000", 2024-09-18T16:16:01.817 INFO:tasks.workunit.client.0.smithi046.stdout: "lost": 0, 2024-09-18T16:16:01.817 INFO:tasks.workunit.client.0.smithi046.stdout: "flags": [ 2024-09-18T16:16:01.817 INFO:tasks.workunit.client.0.smithi046.stdout: "dirty", 2024-09-18T16:16:01.817 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest" 2024-09-18T16:16:01.817 INFO:tasks.workunit.client.0.smithi046.stdout: ], 2024-09-18T16:16:01.817 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_seq": 0, 2024-09-18T16:16:01.817 INFO:tasks.workunit.client.0.smithi046.stdout: "truncate_size": 0, 2024-09-18T16:16:01.817 INFO:tasks.workunit.client.0.smithi046.stdout: "data_digest": "0x485963c8", 2024-09-18T16:16:01.817 INFO:tasks.workunit.client.0.smithi046.stdout: "omap_digest": "0xffffffff", 2024-09-18T16:16:01.817 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_object_size": 0, 2024-09-18T16:16:01.817 INFO:tasks.workunit.client.0.smithi046.stdout: "expected_write_size": 0, 2024-09-18T16:16:01.817 INFO:tasks.workunit.client.0.smithi046.stdout: "alloc_hint_flags": 0, 2024-09-18T16:16:01.818 INFO:tasks.workunit.client.0.smithi046.stdout: "manifest": { 2024-09-18T16:16:01.818 INFO:tasks.workunit.client.0.smithi046.stdout: "type": 0 2024-09-18T16:16:01.818 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.818 INFO:tasks.workunit.client.0.smithi046.stdout: "watchers": {} 2024-09-18T16:16:01.818 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.818 INFO:tasks.workunit.client.0.smithi046.stdout: "shards": [ 2024-09-18T16:16:01.818 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.818 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 0, 2024-09-18T16:16:01.818 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": false, 2024-09-18T16:16:01.818 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.818 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:16:01.818 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:16:01.818 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:16:01.819 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:16:01.819 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.819 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:16:01.819 INFO:tasks.workunit.client.0.smithi046.stdout: "size": "????", 2024-09-18T16:16:01.819 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]", 2024-09-18T16:16:01.819 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:16:01.820 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T16:16:01.820 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.820 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.820 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.820 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.820 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-09-18T16:16:01.820 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.820 INFO:tasks.workunit.client.0.smithi046.stdout: "osd": 1, 2024-09-18T16:16:01.820 INFO:tasks.workunit.client.0.smithi046.stdout: "primary": true, 2024-09-18T16:16:01.820 INFO:tasks.workunit.client.0.smithi046.stdout: "errors": [], 2024-09-18T16:16:01.820 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 256, 2024-09-18T16:16:01.820 INFO:tasks.workunit.client.0.smithi046.stdout: "snapset": { 2024-09-18T16:16:01.820 INFO:tasks.workunit.client.0.smithi046.stdout: "seq": 1, 2024-09-18T16:16:01.821 INFO:tasks.workunit.client.0.smithi046.stdout: "clones": [ 2024-09-18T16:16:01.821 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-09-18T16:16:01.821 INFO:tasks.workunit.client.0.smithi046.stdout: "snap": 1, 2024-09-18T16:16:01.821 INFO:tasks.workunit.client.0.smithi046.stdout: "size": 1032, 2024-09-18T16:16:01.821 INFO:tasks.workunit.client.0.smithi046.stdout: "overlap": "[]", 2024-09-18T16:16:01.821 INFO:tasks.workunit.client.0.smithi046.stdout: "snaps": [ 2024-09-18T16:16:01.821 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2024-09-18T16:16:01.821 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.821 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.821 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.821 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.821 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.821 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.822 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-09-18T16:16:01.822 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-09-18T16:16:01.822 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:16:01.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:780: _scrub_snaps_multi: rados list-inconsistent-snapset 1.0 2024-09-18T16:16:01.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:783: _scrub_snaps_multi: '[' replica = replica ']' 2024-09-18T16:16:01.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:785: _scrub_snaps_multi: scruberrors=20 2024-09-18T16:16:01.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: jq .inconsistents 2024-09-18T16:16:01.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-18T16:16:01.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1049: _scrub_snaps_multi: jq .inconsistents td/osd-scrub-snaps/json 2024-09-18T16:16:01.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1049: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-18T16:16:01.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1050: _scrub_snaps_multi: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-18T16:16:01.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-18T16:16:01.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1051: _scrub_snaps_multi: test no = yes 2024-09-18T16:16:01.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1056: _scrub_snaps_multi: test '' = yes 2024-09-18T16:16:01.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1061: _scrub_snaps_multi: find td/osd-scrub-snaps 2024-09-18T16:16:01.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1061: _scrub_snaps_multi: grep 'osd[^/]*\.pid' 2024-09-18T16:16:01.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1061: _scrub_snaps_multi: pidfiles='td/osd-scrub-snaps/osd.0.pid 2024-09-18T16:16:01.845 INFO:tasks.workunit.client.0.smithi046.stderr:td/osd-scrub-snaps/osd.1.pid' 2024-09-18T16:16:01.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1062: _scrub_snaps_multi: pids= 2024-09-18T16:16:01.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1063: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-09-18T16:16:01.845 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1065: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.0.pid 2024-09-18T16:16:01.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1065: _scrub_snaps_multi: pids+='560406 ' 2024-09-18T16:16:01.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1063: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-09-18T16:16:01.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1065: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.1.pid 2024-09-18T16:16:01.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1065: _scrub_snaps_multi: pids+='560984 ' 2024-09-18T16:16:01.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1068: _scrub_snaps_multi: ERRORS=0 2024-09-18T16:16:01.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1072: _scrub_snaps_multi: '[' replica = primary ']' 2024-09-18T16:16:01.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1094: _scrub_snaps_multi: ceph pg dump pgs 2024-09-18T16:16:02.145 INFO:tasks.workunit.client.0.smithi046.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T16:16:02.145 INFO:tasks.workunit.client.0.smithi046.stdout:1.0 36 0 0 0 0 24448 0 0 56 0 56 active+clean+inconsistent 2024-09-18T16:15:58.481320+0000 23'56 30:89 [1,0] 1 [1,0] 1 23'56 2024-09-18T16:15:58.481142+0000 0'0 2024-09-18T16:14:28.082713+0000 0 1 periodic scrub scheduled @ 2024-09-19T20:17:09.629090+0000 36 0 2024-09-18T16:16:02.145 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-09-18T16:16:02.145 INFO:tasks.workunit.client.0.smithi046.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T16:16:02.149 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:16:02.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1096: _scrub_snaps_multi: for pid in $pids 2024-09-18T16:16:02.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1098: _scrub_snaps_multi: kill -0 560406 2024-09-18T16:16:02.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1096: _scrub_snaps_multi: for pid in $pids 2024-09-18T16:16:02.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1098: _scrub_snaps_multi: kill -0 560984 2024-09-18T16:16:02.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1105: _scrub_snaps_multi: kill_daemons td/osd-scrub-snaps 2024-09-18T16:16:02.159 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:16:02.159 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:16:02.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:16:02.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:16:02.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:16:07.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:16:07.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1107: _scrub_snaps_multi: declare -a err_strings 2024-09-18T16:16:07.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1108: _scrub_snaps_multi: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' 2024-09-18T16:16:07.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1109: _scrub_snaps_multi: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' 2024-09-18T16:16:07.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1110: _scrub_snaps_multi: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' 2024-09-18T16:16:07.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1111: _scrub_snaps_multi: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' 2024-09-18T16:16:07.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1112: _scrub_snaps_multi: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' 2024-09-18T16:16:07.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' 2024-09-18T16:16:07.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1114: _scrub_snaps_multi: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' 2024-09-18T16:16:07.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' 2024-09-18T16:16:07.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1117: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-18T16:16:07.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1119: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' td/osd-scrub-snaps/osd.1.log 2024-09-18T16:16:07.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1117: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-18T16:16:07.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1119: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-09-18T16:16:07.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1117: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-18T16:16:07.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1119: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' td/osd-scrub-snaps/osd.1.log 2024-09-18T16:16:07.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1117: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-18T16:16:07.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1119: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' td/osd-scrub-snaps/osd.1.log 2024-09-18T16:16:07.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1117: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-18T16:16:07.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1119: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-09-18T16:16:07.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1117: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-18T16:16:07.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1119: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' td/osd-scrub-snaps/osd.1.log 2024-09-18T16:16:07.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1117: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-18T16:16:07.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1119: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' td/osd-scrub-snaps/osd.1.log 2024-09-18T16:16:07.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1117: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-18T16:16:07.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1119: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' td/osd-scrub-snaps/osd.1.log 2024-09-18T16:16:07.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1127: _scrub_snaps_multi: declare -a rep_err_strings 2024-09-18T16:16:07.524 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1128: _scrub_snaps_multi: eval echo '$replica' 2024-09-18T16:16:07.524 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1128: _scrub_snaps_multi: echo 0 2024-09-18T16:16:07.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1128: _scrub_snaps_multi: osd=0 2024-09-18T16:16:07.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1129: _scrub_snaps_multi: rep_err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2024-09-18T16:16:07.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1130: _scrub_snaps_multi: for err_string in "${rep_err_strings[@]}" 2024-09-18T16:16:07.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1132: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.0.log 2024-09-18T16:16:07.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1139: _scrub_snaps_multi: '[' 0 '!=' 0 ']' 2024-09-18T16:16:07.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: echo 'TEST PASSED' 2024-09-18T16:16:07.530 INFO:tasks.workunit.client.0.smithi046.stdout:TEST PASSED 2024-09-18T16:16:07.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1146: _scrub_snaps_multi: return 0 2024-09-18T16:16:07.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1154: TEST_scrub_snaps_replica: err=0 2024-09-18T16:16:07.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1155: TEST_scrub_snaps_replica: CEPH_ARGS='--fsid=89666f4e-d227-401b-a2df-60eab9d6e1bb --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-09-18T16:16:07.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1156: TEST_scrub_snaps_replica: return 0 2024-09-18T16:16:07.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2024-09-18T16:16:07.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-09-18T16:16:07.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:16:07.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-18T16:16:07.533 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:16:07.533 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:16:07.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:16:07.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:16:07.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:16:07.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:16:07.545 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:16:07.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:16:07.546 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:16:07.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:16:07.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:16:07.549 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:16:07.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:16:07.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:16:07.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:16:07.551 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:16:07.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:16:07.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:16:07.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-09-18T16:16:07.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:16:07.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:16:07.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:16:07.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.495962 2024-09-18T16:16:07.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:16:07.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:16:07.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T16:16:07.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-scrub-snaps 0 2024-09-18T16:16:07.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-09-18T16:16:07.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T16:16:07.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-18T16:16:07.576 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:16:07.576 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:16:07.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:16:07.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:16:07.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:16:07.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:16:07.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:16:07.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:16:07.583 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:16:07.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:16:07.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:16:07.586 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:16:07.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:16:07.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:16:07.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:16:07.589 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:16:07.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:16:07.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T16:16:07.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-09-18T16:16:07.595 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:16:07.595 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:16:07.595 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.495962 2024-09-18T16:16:07.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.495962 2024-09-18T16:16:07.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:16:07.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:16:07.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T16:16:07.599 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T16:16:07.599 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T16:16:07.655 INFO:tasks.workunit:Running workunit scrub/osd-scrub-test.sh... 2024-09-18T16:16:07.656 DEBUG:teuthology.orchestra.run.smithi046:workunit test scrub/osd-scrub-test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh 2024-09-18T16:16:07.675 INFO:tasks.workunit.client.0.smithi046.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T16:16:07.690 INFO:tasks.workunit.client.0.smithi046.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T16:16:07.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-18T16:16:07.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-18T16:16:07.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T16:16:07.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T16:16:07.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T16:16:07.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T16:16:07.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T16:16:07.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T16:16:07.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-scrub-test 2024-09-18T16:16:07.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:21: run: local dir=td/osd-scrub-test 2024-09-18T16:16:07.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:22: run: shift 2024-09-18T16:16:07.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:24: run: export CEPH_MON=127.0.0.1:7138 2024-09-18T16:16:07.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:24: run: CEPH_MON=127.0.0.1:7138 2024-09-18T16:16:07.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:25: run: export CEPH_ARGS 2024-09-18T16:16:07.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:26: run: uuidgen 2024-09-18T16:16:07.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:26: run: CEPH_ARGS+='--fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none ' 2024-09-18T16:16:07.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7138 ' 2024-09-18T16:16:07.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-09-18T16:16:07.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: set 2024-09-18T16:16:07.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T16:16:07.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: local 'funcs=TEST_deep_scrub_abort 2024-09-18T16:16:07.697 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_dump_scrub_schedule 2024-09-18T16:16:07.697 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_interval_changes 2024-09-18T16:16:07.697 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_just_deep_scrubs 2024-09-18T16:16:07.697 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_pg_dump_objects_scrubbed 2024-09-18T16:16:07.697 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_scrub_abort 2024-09-18T16:16:07.697 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_scrub_extended_sleep 2024-09-18T16:16:07.697 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_scrub_permit_time 2024-09-18T16:16:07.697 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_scrub_test' 2024-09-18T16:16:07.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-09-18T16:16:07.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: setup td/osd-scrub-test 2024-09-18T16:16:07.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-test 2024-09-18T16:16:07.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-test 2024-09-18T16:16:07.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-09-18T16:16:07.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:16:07.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-18T16:16:07.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:16:07.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:16:07.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:16:07.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:16:07.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:16:07.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:16:07.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:16:07.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:16:07.703 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:16:07.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:16:07.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:16:07.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:16:07.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:16:07.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:16:07.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:16:07.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:16:07.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:16:07.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:16:07.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-09-18T16:16:07.714 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:16:07.714 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:16:07.714 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:16:07.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.562265 2024-09-18T16:16:07.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:16:07.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:16:07.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-test 2024-09-18T16:16:07.718 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T16:16:07.718 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:16:07.718 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:16:07.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.562265 2024-09-18T16:16:07.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T16:16:07.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T16:16:07.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T16:16:07.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T16:16:07.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-09-18T16:16:07.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: TEST_deep_scrub_abort td/osd-scrub-test 2024-09-18T16:16:07.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:396: TEST_deep_scrub_abort: local dir=td/osd-scrub-test 2024-09-18T16:16:07.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:397: TEST_deep_scrub_abort: _scrub_abort td/osd-scrub-test deep_scrub 2024-09-18T16:16:07.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:276: _scrub_abort: local dir=td/osd-scrub-test 2024-09-18T16:16:07.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:277: _scrub_abort: local poolname=test 2024-09-18T16:16:07.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:278: _scrub_abort: local OSDS=3 2024-09-18T16:16:07.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:279: _scrub_abort: local objects=1000 2024-09-18T16:16:07.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:280: _scrub_abort: local type=deep_scrub 2024-09-18T16:16:07.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:282: _scrub_abort: TESTDATA=testdata.562265 2024-09-18T16:16:07.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:283: _scrub_abort: test deep_scrub = scrub 2024-09-18T16:16:07.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:288: _scrub_abort: stopscrub=nodeep-scrub 2024-09-18T16:16:07.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:289: _scrub_abort: check=nodeep_scrub 2024-09-18T16:16:07.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:292: _scrub_abort: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2024-09-18T16:16:07.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-test 2024-09-18T16:16:07.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T16:16:07.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T16:16:07.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T16:16:07.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-test/a 2024-09-18T16:16:07.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2024-09-18T16:16:08.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T16:16:08.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:16:08.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:16:08.095 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:16:08.095 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:16:08.095 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:16:08.095 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:16:08.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-18T16:16:08.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T16:16:08.378 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T16:16:08.378 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:16:08.378 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:16:08.378 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T16:16:08.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T16:16:08.379 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:16:08.379 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:16:08.380 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:16:08.380 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:16:08.380 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:16:08.380 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:16:08.381 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.562265/ceph-mon.a.asok 2024-09-18T16:16:08.381 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:16:08.381 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.562265/ceph-mon.a.asok config get fsid 2024-09-18T16:16:08.478 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T16:16:08.478 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:16:08.478 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:16:08.478 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T16:16:08.479 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T16:16:08.479 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:16:08.479 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:16:08.480 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:16:08.480 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:16:08.480 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:16:08.480 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:16:08.480 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.562265/ceph-mon.a.asok 2024-09-18T16:16:08.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:16:08.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.562265/ceph-mon.a.asok config get mon_host 2024-09-18T16:16:08.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:293: _scrub_abort: run_mgr td/osd-scrub-test x 2024-09-18T16:16:08.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-test 2024-09-18T16:16:08.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T16:16:08.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T16:16:08.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T16:16:08.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-test/x 2024-09-18T16:16:08.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T16:16:09.419 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T16:16:09.420 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:16:09.420 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:16:09.420 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:16:09.421 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:16:09.421 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:16:09.421 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:16:09.422 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:16:09.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:16:09.455 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:294: _scrub_abort: expr 3 - 1 2024-09-18T16:16:09.457 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:294: _scrub_abort: seq 0 2 2024-09-18T16:16:09.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:294: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:16:09.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:300: _scrub_abort: run_osd td/osd-scrub-test 0 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-18T16:16:09.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-09-18T16:16:09.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:16:09.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T16:16:09.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:16:09.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/0 2024-09-18T16:16:09.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-18T16:16:09.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:16:09.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:16:09.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:16:09.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-09-18T16:16:09.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-09-18T16:16:09.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:16:09.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:16:09.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-18T16:16:09.462 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:16:09.462 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:16:09.462 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:16:09.463 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:16:09.463 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:16:09.463 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:16:09.463 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:16:09.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:16:09.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:16:09.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:16:09.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:16:09.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-18T16:16:09.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-18T16:16:09.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:16:09.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:16:09.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:16:09.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:16:09.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:16:09.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-09-18T16:16:09.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/0 2024-09-18T16:16:09.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:16:09.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ce978e87-8aa9-42fb-901f-df665e53d74a 2024-09-18T16:16:09.467 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 ce978e87-8aa9-42fb-901f-df665e53d74a 2024-09-18T16:16:09.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ce978e87-8aa9-42fb-901f-df665e53d74a' 2024-09-18T16:16:09.468 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:16:09.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDJ/OpmV4HZHBAAOYUIVyNFHBSUM2AVjzXoHw== 2024-09-18T16:16:09.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDJ/OpmV4HZHBAAOYUIVyNFHBSUM2AVjzXoHw=="}' 2024-09-18T16:16:09.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ce978e87-8aa9-42fb-901f-df665e53d74a -i td/osd-scrub-test/0/new.json 2024-09-18T16:16:09.770 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:16:09.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/0/new.json 2024-09-18T16:16:09.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQDJ/OpmV4HZHBAAOYUIVyNFHBSUM2AVjzXoHw== --osd-uuid ce978e87-8aa9-42fb-901f-df665e53d74a 2024-09-18T16:16:09.817 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:09.809+0000 7fa138d9b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:16:09.821 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:09.813+0000 7fa138d9b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:16:09.821 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:09.817+0000 7fa138d9b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:16:09.822 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:09.817+0000 7fa138d9b540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:16:09.822 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:09.817+0000 7fa138d9b540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:16:09.822 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:09.817+0000 7fa138d9b540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:16:09.886 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:09.881+0000 7fa138d9b540 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-09-18T16:16:15.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-09-18T16:16:15.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:16:15.892 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T16:16:15.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T16:16:15.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:16:16.273 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T16:16:16.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T16:16:16.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-18T16:16:16.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:16:16.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:16:16.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:16:16.305 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:16.297+0000 7f1367944540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:16:16.308 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:16.301+0000 7f1367944540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:16:16.309 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:16.301+0000 7f1367944540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:16:16.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T16:16:16.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:16:16.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:16:16.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:16:16.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:16:16.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:16:16.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:16:16.577 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:16:16.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:16:16.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:16:16.623 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:16.617+0000 7f1367944540 -1 Falling back to public interface 2024-09-18T16:16:16.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:16:17.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:16:17.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:16:17.931 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:16:17.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:16:17.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:16:17.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:16:18.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:16:19.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:16:19.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:16:19.245 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:16:19.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:16:19.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:16:19.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:16:19.356 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:19.349+0000 7f1367944540 -1 osd.0 0 log_to_monitors true 2024-09-18T16:16:19.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:16:20.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:16:20.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:16:20.563 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:16:20.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:16:20.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:16:20.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:16:20.943 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/563385,v1:127.0.0.1:6803/563385] [v2:127.0.0.1:6804/563385,v1:127.0.0.1:6805/563385] exists,up ce978e87-8aa9-42fb-901f-df665e53d74a 2024-09-18T16:16:20.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:16:20.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:16:20.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:16:20.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:294: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:16:20.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:300: _scrub_abort: run_osd td/osd-scrub-test 1 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-18T16:16:20.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-09-18T16:16:20.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:16:20.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T16:16:20.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:16:20.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/1 2024-09-18T16:16:20.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-18T16:16:20.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:16:20.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:16:20.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:16:20.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-09-18T16:16:20.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-09-18T16:16:20.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:16:20.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:16:20.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-18T16:16:20.947 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:16:20.947 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:16:20.947 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:16:20.947 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:16:20.947 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:16:20.947 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:16:20.948 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:16:20.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:16:20.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:16:20.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:16:20.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:16:20.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-18T16:16:20.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-18T16:16:20.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:16:20.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:16:20.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:16:20.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:16:20.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:16:20.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-09-18T16:16:20.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/1 2024-09-18T16:16:20.949 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:16:20.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d12858b4-96e0-441f-b954-41b7defeb37d 2024-09-18T16:16:20.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d12858b4-96e0-441f-b954-41b7defeb37d' 2024-09-18T16:16:20.950 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 d12858b4-96e0-441f-b954-41b7defeb37d 2024-09-18T16:16:20.950 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:16:20.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDU/OpmCsegORAAu8LahgMQkMLG7iu+xoKvGg== 2024-09-18T16:16:20.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDU/OpmCsegORAAu8LahgMQkMLG7iu+xoKvGg=="}' 2024-09-18T16:16:20.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d12858b4-96e0-441f-b954-41b7defeb37d -i td/osd-scrub-test/1/new.json 2024-09-18T16:16:21.346 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:16:21.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/1/new.json 2024-09-18T16:16:21.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQDU/OpmCsegORAAu8LahgMQkMLG7iu+xoKvGg== --osd-uuid d12858b4-96e0-441f-b954-41b7defeb37d 2024-09-18T16:16:21.399 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:21.393+0000 7f3d9af79540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:16:21.404 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:21.397+0000 7f3d9af79540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:16:21.406 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:21.401+0000 7f3d9af79540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:16:21.407 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:21.401+0000 7f3d9af79540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:16:21.407 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:21.401+0000 7f3d9af79540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:16:21.407 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:21.401+0000 7f3d9af79540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:16:21.467 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:21.461+0000 7f3d9af79540 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-09-18T16:16:27.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-09-18T16:16:27.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:16:27.672 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T16:16:27.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T16:16:27.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:16:28.050 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T16:16:28.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T16:16:28.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-18T16:16:28.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:16:28.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:16:28.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:16:28.082 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:28.077+0000 7f306fd26540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:16:28.085 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:28.077+0000 7f306fd26540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:16:28.086 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:28.081+0000 7f306fd26540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:16:28.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T16:16:28.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:16:28.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T16:16:28.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:16:28.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:16:28.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:16:28.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:16:28.346 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:16:28.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:16:28.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:16:28.636 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:28.629+0000 7f306fd26540 -1 Falling back to public interface 2024-09-18T16:16:28.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:16:29.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:16:29.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:16:29.698 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:16:29.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:16:29.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:16:29.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:16:30.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:16:31.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:16:31.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:16:31.005 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:16:31.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:16:31.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:16:31.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:16:31.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:16:31.448 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:31.441+0000 7f306fd26540 -1 osd.1 0 log_to_monitors true 2024-09-18T16:16:32.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:16:32.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:16:32.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:16:32.318 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:16:32.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:16:32.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:16:32.660 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/564784,v1:127.0.0.1:6811/564784] [v2:127.0.0.1:6812/564784,v1:127.0.0.1:6813/564784] exists,up d12858b4-96e0-441f-b954-41b7defeb37d 2024-09-18T16:16:32.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:16:32.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:16:32.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:16:32.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:294: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:16:32.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:300: _scrub_abort: run_osd td/osd-scrub-test 2 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-18T16:16:32.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-09-18T16:16:32.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:16:32.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T16:16:32.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:16:32.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/2 2024-09-18T16:16:32.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-18T16:16:32.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:16:32.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:16:32.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:16:32.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2024-09-18T16:16:32.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2024-09-18T16:16:32.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:16:32.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:16:32.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-18T16:16:32.664 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:16:32.664 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:16:32.664 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:16:32.666 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:16:32.666 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:16:32.666 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:16:32.666 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:16:32.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:16:32.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:16:32.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:16:32.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:16:32.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-18T16:16:32.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-18T16:16:32.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:16:32.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:16:32.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:16:32.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:16:32.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:16:32.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-09-18T16:16:32.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/2 2024-09-18T16:16:32.667 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:16:32.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=da1fbdeb-b18a-4f97-bb56-c86757f3ee99 2024-09-18T16:16:32.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 da1fbdeb-b18a-4f97-bb56-c86757f3ee99' 2024-09-18T16:16:32.669 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 da1fbdeb-b18a-4f97-bb56-c86757f3ee99 2024-09-18T16:16:32.669 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:16:32.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDg/OpmO7HHKBAAWrq71ET5FIwuyEoum8cVbw== 2024-09-18T16:16:32.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDg/OpmO7HHKBAAWrq71ET5FIwuyEoum8cVbw=="}' 2024-09-18T16:16:32.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new da1fbdeb-b18a-4f97-bb56-c86757f3ee99 -i td/osd-scrub-test/2/new.json 2024-09-18T16:16:33.074 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:16:33.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/2/new.json 2024-09-18T16:16:33.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQDg/OpmO7HHKBAAWrq71ET5FIwuyEoum8cVbw== --osd-uuid da1fbdeb-b18a-4f97-bb56-c86757f3ee99 2024-09-18T16:16:33.125 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:33.121+0000 7fde344a6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:16:33.130 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:33.125+0000 7fde344a6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:16:33.132 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:33.125+0000 7fde344a6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:16:33.132 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:33.125+0000 7fde344a6540 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-09-18T16:16:33.132 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:33.125+0000 7fde344a6540 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-09-18T16:16:33.132 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:33.125+0000 7fde344a6540 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-09-18T16:16:33.213 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:33.209+0000 7fde344a6540 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-09-18T16:16:39.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2024-09-18T16:16:39.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:16:39.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T16:16:39.490 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T16:16:39.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:16:39.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T16:16:39.888 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T16:16:39.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-18T16:16:39.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:16:39.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:16:39.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:16:39.920 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:39.913+0000 7f27375a6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:16:39.923 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:39.917+0000 7f27375a6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:16:39.929 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:39.917+0000 7f27375a6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:16:40.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T16:16:40.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:16:40.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T16:16:40.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:16:40.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:16:40.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:16:40.222 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:16:40.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:16:40.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:16:40.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:16:40.467 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:40.461+0000 7f27375a6540 -1 Falling back to public interface 2024-09-18T16:16:40.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:16:41.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:16:41.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:16:41.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:16:41.575 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:16:41.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:16:41.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:16:41.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:16:42.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:16:42.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:16:42.893 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:16:42.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:16:42.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:16:42.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:16:43.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:16:43.358 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:16:43.353+0000 7f27375a6540 -1 osd.2 0 log_to_monitors true 2024-09-18T16:16:44.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:16:44.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:16:44.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:16:44.203 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:16:44.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:16:44.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:16:44.563 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/566183,v1:127.0.0.1:6819/566183] [v2:127.0.0.1:6820/566183,v1:127.0.0.1:6821/566183] exists,up da1fbdeb-b18a-4f97-bb56-c86757f3ee99 2024-09-18T16:16:44.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:16:44.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:16:44.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:16:44.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:308: _scrub_abort: create_pool test 1 1 2024-09-18T16:16:44.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-18T16:16:45.060 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'test' created 2024-09-18T16:16:45.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T16:16:46.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:309: _scrub_abort: wait_for_clean 2024-09-18T16:16:46.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:16:46.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:16:46.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:16:46.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:16:46.076 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:16:46.077 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:16:46.077 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:16:46.078 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:16:46.078 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:16:46.078 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:16:46.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:16:46.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:16:46.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:16:46.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:16:46.294 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:16:46.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:16:46.604 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:16:46.604 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:16:46.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:16:46.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:16:46.605 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:16:46.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836487 2024-09-18T16:16:46.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836487 2024-09-18T16:16:46.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836487' 2024-09-18T16:16:46.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:16:46.748 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:16:46.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672964 2024-09-18T16:16:46.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672964 2024-09-18T16:16:46.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836487 1-42949672964' 2024-09-18T16:16:46.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:16:46.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:16:47.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509442 2024-09-18T16:16:47.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509442 2024-09-18T16:16:47.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836487 1-42949672964 2-64424509442' 2024-09-18T16:16:47.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:16:47.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836487 2024-09-18T16:16:47.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:16:47.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:16:47.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836487 2024-09-18T16:16:47.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:16:47.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836487 2024-09-18T16:16:47.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-09-18T16:16:47.058 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836487 2024-09-18T16:16:47.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:16:47.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836487 -lt 21474836487 2024-09-18T16:16:47.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:16:47.364 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672964 2024-09-18T16:16:47.364 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:16:47.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:16:47.367 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672964 2024-09-18T16:16:47.367 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:16:47.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672964 2024-09-18T16:16:47.369 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672964 2024-09-18T16:16:47.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-09-18T16:16:47.370 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:16:47.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672963 -lt 42949672964 2024-09-18T16:16:47.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:16:48.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:16:48.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:16:49.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672965 -lt 42949672964 2024-09-18T16:16:49.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:16:49.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509442 2024-09-18T16:16:49.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:16:49.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:16:49.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509442 2024-09-18T16:16:49.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:16:49.062 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509442 2024-09-18T16:16:49.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509442 2024-09-18T16:16:49.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-09-18T16:16:49.063 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:16:49.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509442 -lt 64424509442 2024-09-18T16:16:49.412 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:16:49.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:16:49.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:16:49.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:16:49.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:16:49.783 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:16:49.783 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:16:49.783 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:16:49.783 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:16:49.784 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:16:49.784 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:16:50.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:16:50.120 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:16:50.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:16:50.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:16:50.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:16:50.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:16:50.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:16:50.478 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:310: _scrub_abort: ceph osd dump 2024-09-18T16:16:50.478 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:310: _scrub_abort: grep '^pool.*['\'']test['\'']' 2024-09-18T16:16:50.478 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:310: _scrub_abort: awk '{ print $2 }' 2024-09-18T16:16:50.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:310: _scrub_abort: poolid=1 2024-09-18T16:16:50.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:312: _scrub_abort: dd if=/dev/urandom of=testdata.562265 bs=1032 count=1 2024-09-18T16:16:50.814 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records in 2024-09-18T16:16:50.814 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records out 2024-09-18T16:16:50.814 INFO:tasks.workunit.client.0.smithi046.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 0.000184282 s, 5.6 MB/s 2024-09-18T16:16:50.815 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: seq 1 1000 2024-09-18T16:16:50.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:50.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj1 testdata.562265 2024-09-18T16:16:50.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:50.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj2 testdata.562265 2024-09-18T16:16:51.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:51.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj3 testdata.562265 2024-09-18T16:16:51.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:51.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj4 testdata.562265 2024-09-18T16:16:51.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:51.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj5 testdata.562265 2024-09-18T16:16:51.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:51.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj6 testdata.562265 2024-09-18T16:16:51.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:51.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj7 testdata.562265 2024-09-18T16:16:51.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:51.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj8 testdata.562265 2024-09-18T16:16:52.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:52.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj9 testdata.562265 2024-09-18T16:16:52.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:52.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj10 testdata.562265 2024-09-18T16:16:52.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:52.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj11 testdata.562265 2024-09-18T16:16:52.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:52.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj12 testdata.562265 2024-09-18T16:16:52.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:52.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj13 testdata.562265 2024-09-18T16:16:52.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:52.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj14 testdata.562265 2024-09-18T16:16:53.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:53.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj15 testdata.562265 2024-09-18T16:16:53.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:53.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj16 testdata.562265 2024-09-18T16:16:53.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:53.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj17 testdata.562265 2024-09-18T16:16:53.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:53.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj18 testdata.562265 2024-09-18T16:16:53.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:53.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj19 testdata.562265 2024-09-18T16:16:53.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:53.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj20 testdata.562265 2024-09-18T16:16:53.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:53.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj21 testdata.562265 2024-09-18T16:16:54.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:54.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj22 testdata.562265 2024-09-18T16:16:54.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:54.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj23 testdata.562265 2024-09-18T16:16:54.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:54.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj24 testdata.562265 2024-09-18T16:16:54.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:54.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj25 testdata.562265 2024-09-18T16:16:54.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:54.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj26 testdata.562265 2024-09-18T16:16:54.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:54.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj27 testdata.562265 2024-09-18T16:16:54.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:54.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj28 testdata.562265 2024-09-18T16:16:55.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:55.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj29 testdata.562265 2024-09-18T16:16:55.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:55.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj30 testdata.562265 2024-09-18T16:16:55.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:55.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj31 testdata.562265 2024-09-18T16:16:55.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:55.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj32 testdata.562265 2024-09-18T16:16:55.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:55.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj33 testdata.562265 2024-09-18T16:16:55.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:55.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj34 testdata.562265 2024-09-18T16:16:56.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:56.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj35 testdata.562265 2024-09-18T16:16:56.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:56.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj36 testdata.562265 2024-09-18T16:16:56.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:56.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj37 testdata.562265 2024-09-18T16:16:56.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:56.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj38 testdata.562265 2024-09-18T16:16:56.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:56.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj39 testdata.562265 2024-09-18T16:16:56.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:56.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj40 testdata.562265 2024-09-18T16:16:56.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:56.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj41 testdata.562265 2024-09-18T16:16:57.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:57.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj42 testdata.562265 2024-09-18T16:16:57.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:57.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj43 testdata.562265 2024-09-18T16:16:57.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:57.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj44 testdata.562265 2024-09-18T16:16:57.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:57.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj45 testdata.562265 2024-09-18T16:16:57.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:57.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj46 testdata.562265 2024-09-18T16:16:57.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:57.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj47 testdata.562265 2024-09-18T16:16:57.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:57.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj48 testdata.562265 2024-09-18T16:16:58.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:58.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj49 testdata.562265 2024-09-18T16:16:58.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:58.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj50 testdata.562265 2024-09-18T16:16:58.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:58.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj51 testdata.562265 2024-09-18T16:16:58.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:58.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj52 testdata.562265 2024-09-18T16:16:58.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:58.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj53 testdata.562265 2024-09-18T16:16:58.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:58.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj54 testdata.562265 2024-09-18T16:16:58.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:58.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj55 testdata.562265 2024-09-18T16:16:58.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:58.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj56 testdata.562265 2024-09-18T16:16:59.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:59.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj57 testdata.562265 2024-09-18T16:16:59.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:59.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj58 testdata.562265 2024-09-18T16:16:59.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:59.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj59 testdata.562265 2024-09-18T16:16:59.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:59.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj60 testdata.562265 2024-09-18T16:16:59.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:59.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj61 testdata.562265 2024-09-18T16:16:59.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:59.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj62 testdata.562265 2024-09-18T16:16:59.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:16:59.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj63 testdata.562265 2024-09-18T16:17:00.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:00.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj64 testdata.562265 2024-09-18T16:17:00.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:00.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj65 testdata.562265 2024-09-18T16:17:00.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:00.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj66 testdata.562265 2024-09-18T16:17:00.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:00.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj67 testdata.562265 2024-09-18T16:17:00.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:00.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj68 testdata.562265 2024-09-18T16:17:01.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:01.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj69 testdata.562265 2024-09-18T16:17:01.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:01.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj70 testdata.562265 2024-09-18T16:17:01.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:01.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj71 testdata.562265 2024-09-18T16:17:01.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:01.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj72 testdata.562265 2024-09-18T16:17:01.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:01.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj73 testdata.562265 2024-09-18T16:17:01.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:01.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj74 testdata.562265 2024-09-18T16:17:01.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:01.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj75 testdata.562265 2024-09-18T16:17:02.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:02.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj76 testdata.562265 2024-09-18T16:17:02.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:02.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj77 testdata.562265 2024-09-18T16:17:02.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:02.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj78 testdata.562265 2024-09-18T16:17:02.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:02.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj79 testdata.562265 2024-09-18T16:17:02.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:02.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj80 testdata.562265 2024-09-18T16:17:02.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:02.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj81 testdata.562265 2024-09-18T16:17:02.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:02.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj82 testdata.562265 2024-09-18T16:17:03.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:03.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj83 testdata.562265 2024-09-18T16:17:03.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:03.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj84 testdata.562265 2024-09-18T16:17:03.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:03.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj85 testdata.562265 2024-09-18T16:17:03.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:03.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj86 testdata.562265 2024-09-18T16:17:03.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:03.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj87 testdata.562265 2024-09-18T16:17:03.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:03.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj88 testdata.562265 2024-09-18T16:17:04.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:04.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj89 testdata.562265 2024-09-18T16:17:04.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:04.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj90 testdata.562265 2024-09-18T16:17:04.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:04.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj91 testdata.562265 2024-09-18T16:17:04.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:04.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj92 testdata.562265 2024-09-18T16:17:04.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:04.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj93 testdata.562265 2024-09-18T16:17:04.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:04.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj94 testdata.562265 2024-09-18T16:17:04.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:04.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj95 testdata.562265 2024-09-18T16:17:05.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:05.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj96 testdata.562265 2024-09-18T16:17:05.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:05.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj97 testdata.562265 2024-09-18T16:17:05.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:05.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj98 testdata.562265 2024-09-18T16:17:05.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:05.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj99 testdata.562265 2024-09-18T16:17:05.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:05.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj100 testdata.562265 2024-09-18T16:17:05.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:05.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj101 testdata.562265 2024-09-18T16:17:05.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:05.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj102 testdata.562265 2024-09-18T16:17:06.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:06.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj103 testdata.562265 2024-09-18T16:17:06.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:06.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj104 testdata.562265 2024-09-18T16:17:06.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:06.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj105 testdata.562265 2024-09-18T16:17:06.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:06.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj106 testdata.562265 2024-09-18T16:17:06.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:06.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj107 testdata.562265 2024-09-18T16:17:06.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:06.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj108 testdata.562265 2024-09-18T16:17:06.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:06.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj109 testdata.562265 2024-09-18T16:17:07.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:07.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj110 testdata.562265 2024-09-18T16:17:07.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:07.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj111 testdata.562265 2024-09-18T16:17:07.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:07.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj112 testdata.562265 2024-09-18T16:17:07.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:07.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj113 testdata.562265 2024-09-18T16:17:07.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:07.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj114 testdata.562265 2024-09-18T16:17:07.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:07.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj115 testdata.562265 2024-09-18T16:17:07.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:07.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj116 testdata.562265 2024-09-18T16:17:08.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:08.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj117 testdata.562265 2024-09-18T16:17:08.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:08.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj118 testdata.562265 2024-09-18T16:17:08.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:08.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj119 testdata.562265 2024-09-18T16:17:08.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:08.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj120 testdata.562265 2024-09-18T16:17:08.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:08.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj121 testdata.562265 2024-09-18T16:17:08.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:08.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj122 testdata.562265 2024-09-18T16:17:08.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:08.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj123 testdata.562265 2024-09-18T16:17:09.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:09.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj124 testdata.562265 2024-09-18T16:17:09.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:09.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj125 testdata.562265 2024-09-18T16:17:09.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:09.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj126 testdata.562265 2024-09-18T16:17:09.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:09.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj127 testdata.562265 2024-09-18T16:17:09.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:09.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj128 testdata.562265 2024-09-18T16:17:09.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:09.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj129 testdata.562265 2024-09-18T16:17:09.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:09.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj130 testdata.562265 2024-09-18T16:17:10.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:10.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj131 testdata.562265 2024-09-18T16:17:10.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:10.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj132 testdata.562265 2024-09-18T16:17:10.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:10.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj133 testdata.562265 2024-09-18T16:17:10.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:10.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj134 testdata.562265 2024-09-18T16:17:10.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:10.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj135 testdata.562265 2024-09-18T16:17:10.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:10.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj136 testdata.562265 2024-09-18T16:17:10.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:10.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj137 testdata.562265 2024-09-18T16:17:11.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:11.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj138 testdata.562265 2024-09-18T16:17:11.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:11.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj139 testdata.562265 2024-09-18T16:17:11.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:11.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj140 testdata.562265 2024-09-18T16:17:11.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:11.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj141 testdata.562265 2024-09-18T16:17:11.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:11.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj142 testdata.562265 2024-09-18T16:17:11.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:11.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj143 testdata.562265 2024-09-18T16:17:11.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:11.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj144 testdata.562265 2024-09-18T16:17:12.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:12.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj145 testdata.562265 2024-09-18T16:17:12.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:12.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj146 testdata.562265 2024-09-18T16:17:12.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:12.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj147 testdata.562265 2024-09-18T16:17:12.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:12.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj148 testdata.562265 2024-09-18T16:17:12.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:12.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj149 testdata.562265 2024-09-18T16:17:12.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:12.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj150 testdata.562265 2024-09-18T16:17:12.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:12.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj151 testdata.562265 2024-09-18T16:17:13.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:13.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj152 testdata.562265 2024-09-18T16:17:13.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:13.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj153 testdata.562265 2024-09-18T16:17:13.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:13.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj154 testdata.562265 2024-09-18T16:17:13.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:13.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj155 testdata.562265 2024-09-18T16:17:13.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:13.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj156 testdata.562265 2024-09-18T16:17:13.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:13.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj157 testdata.562265 2024-09-18T16:17:14.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:14.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj158 testdata.562265 2024-09-18T16:17:14.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:14.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj159 testdata.562265 2024-09-18T16:17:14.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:14.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj160 testdata.562265 2024-09-18T16:17:14.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:14.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj161 testdata.562265 2024-09-18T16:17:14.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:14.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj162 testdata.562265 2024-09-18T16:17:14.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:14.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj163 testdata.562265 2024-09-18T16:17:14.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:14.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj164 testdata.562265 2024-09-18T16:17:15.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:15.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj165 testdata.562265 2024-09-18T16:17:15.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:15.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj166 testdata.562265 2024-09-18T16:17:15.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:15.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj167 testdata.562265 2024-09-18T16:17:15.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:15.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj168 testdata.562265 2024-09-18T16:17:15.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:15.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj169 testdata.562265 2024-09-18T16:17:15.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:15.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj170 testdata.562265 2024-09-18T16:17:16.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:16.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj171 testdata.562265 2024-09-18T16:17:16.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:16.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj172 testdata.562265 2024-09-18T16:17:16.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:16.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj173 testdata.562265 2024-09-18T16:17:16.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:16.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj174 testdata.562265 2024-09-18T16:17:16.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:16.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj175 testdata.562265 2024-09-18T16:17:16.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:16.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj176 testdata.562265 2024-09-18T16:17:16.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:16.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj177 testdata.562265 2024-09-18T16:17:17.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:17.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj178 testdata.562265 2024-09-18T16:17:17.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:17.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj179 testdata.562265 2024-09-18T16:17:17.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:17.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj180 testdata.562265 2024-09-18T16:17:17.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:17.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj181 testdata.562265 2024-09-18T16:17:17.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:17.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj182 testdata.562265 2024-09-18T16:17:17.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:17.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj183 testdata.562265 2024-09-18T16:17:17.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:17.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj184 testdata.562265 2024-09-18T16:17:18.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:18.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj185 testdata.562265 2024-09-18T16:17:18.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:18.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj186 testdata.562265 2024-09-18T16:17:18.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:18.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj187 testdata.562265 2024-09-18T16:17:18.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:18.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj188 testdata.562265 2024-09-18T16:17:18.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:18.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj189 testdata.562265 2024-09-18T16:17:18.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:18.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj190 testdata.562265 2024-09-18T16:17:18.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:18.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj191 testdata.562265 2024-09-18T16:17:18.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:18.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj192 testdata.562265 2024-09-18T16:17:19.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:19.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj193 testdata.562265 2024-09-18T16:17:19.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:19.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj194 testdata.562265 2024-09-18T16:17:19.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:19.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj195 testdata.562265 2024-09-18T16:17:19.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:19.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj196 testdata.562265 2024-09-18T16:17:19.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:19.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj197 testdata.562265 2024-09-18T16:17:19.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:19.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj198 testdata.562265 2024-09-18T16:17:19.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:19.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj199 testdata.562265 2024-09-18T16:17:20.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:20.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj200 testdata.562265 2024-09-18T16:17:20.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:20.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj201 testdata.562265 2024-09-18T16:17:20.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:20.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj202 testdata.562265 2024-09-18T16:17:20.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:20.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj203 testdata.562265 2024-09-18T16:17:20.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:20.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj204 testdata.562265 2024-09-18T16:17:20.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:20.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj205 testdata.562265 2024-09-18T16:17:21.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:21.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj206 testdata.562265 2024-09-18T16:17:21.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:21.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj207 testdata.562265 2024-09-18T16:17:21.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:21.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj208 testdata.562265 2024-09-18T16:17:21.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:21.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj209 testdata.562265 2024-09-18T16:17:21.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:21.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj210 testdata.562265 2024-09-18T16:17:21.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:21.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj211 testdata.562265 2024-09-18T16:17:21.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:21.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj212 testdata.562265 2024-09-18T16:17:22.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:22.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj213 testdata.562265 2024-09-18T16:17:22.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:22.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj214 testdata.562265 2024-09-18T16:17:22.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:22.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj215 testdata.562265 2024-09-18T16:17:22.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:22.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj216 testdata.562265 2024-09-18T16:17:22.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:22.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj217 testdata.562265 2024-09-18T16:17:22.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:22.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj218 testdata.562265 2024-09-18T16:17:22.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:22.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj219 testdata.562265 2024-09-18T16:17:23.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:23.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj220 testdata.562265 2024-09-18T16:17:23.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:23.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj221 testdata.562265 2024-09-18T16:17:23.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:23.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj222 testdata.562265 2024-09-18T16:17:23.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:23.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj223 testdata.562265 2024-09-18T16:17:23.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:23.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj224 testdata.562265 2024-09-18T16:17:23.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:23.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj225 testdata.562265 2024-09-18T16:17:23.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:23.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj226 testdata.562265 2024-09-18T16:17:24.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:24.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj227 testdata.562265 2024-09-18T16:17:24.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:24.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj228 testdata.562265 2024-09-18T16:17:24.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:24.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj229 testdata.562265 2024-09-18T16:17:24.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:24.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj230 testdata.562265 2024-09-18T16:17:24.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:24.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj231 testdata.562265 2024-09-18T16:17:24.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:24.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj232 testdata.562265 2024-09-18T16:17:24.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:24.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj233 testdata.562265 2024-09-18T16:17:25.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:25.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj234 testdata.562265 2024-09-18T16:17:25.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:25.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj235 testdata.562265 2024-09-18T16:17:25.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:25.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj236 testdata.562265 2024-09-18T16:17:25.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:25.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj237 testdata.562265 2024-09-18T16:17:25.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:25.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj238 testdata.562265 2024-09-18T16:17:25.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:25.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj239 testdata.562265 2024-09-18T16:17:26.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:26.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj240 testdata.562265 2024-09-18T16:17:26.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:26.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj241 testdata.562265 2024-09-18T16:17:26.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:26.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj242 testdata.562265 2024-09-18T16:17:26.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:26.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj243 testdata.562265 2024-09-18T16:17:26.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:26.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj244 testdata.562265 2024-09-18T16:17:26.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:26.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj245 testdata.562265 2024-09-18T16:17:26.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:26.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj246 testdata.562265 2024-09-18T16:17:27.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:27.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj247 testdata.562265 2024-09-18T16:17:27.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:27.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj248 testdata.562265 2024-09-18T16:17:27.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:27.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj249 testdata.562265 2024-09-18T16:17:27.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:27.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj250 testdata.562265 2024-09-18T16:17:27.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:27.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj251 testdata.562265 2024-09-18T16:17:27.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:27.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj252 testdata.562265 2024-09-18T16:17:27.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:27.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj253 testdata.562265 2024-09-18T16:17:28.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:28.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj254 testdata.562265 2024-09-18T16:17:28.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:28.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj255 testdata.562265 2024-09-18T16:17:28.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:28.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj256 testdata.562265 2024-09-18T16:17:28.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:28.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj257 testdata.562265 2024-09-18T16:17:28.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:28.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj258 testdata.562265 2024-09-18T16:17:28.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:28.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj259 testdata.562265 2024-09-18T16:17:29.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:29.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj260 testdata.562265 2024-09-18T16:17:29.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:29.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj261 testdata.562265 2024-09-18T16:17:29.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:29.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj262 testdata.562265 2024-09-18T16:17:29.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:29.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj263 testdata.562265 2024-09-18T16:17:29.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:29.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj264 testdata.562265 2024-09-18T16:17:29.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:29.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj265 testdata.562265 2024-09-18T16:17:29.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:29.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj266 testdata.562265 2024-09-18T16:17:30.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:30.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj267 testdata.562265 2024-09-18T16:17:30.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:30.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj268 testdata.562265 2024-09-18T16:17:30.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:30.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj269 testdata.562265 2024-09-18T16:17:30.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:30.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj270 testdata.562265 2024-09-18T16:17:30.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:30.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj271 testdata.562265 2024-09-18T16:17:30.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:30.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj272 testdata.562265 2024-09-18T16:17:31.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:31.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj273 testdata.562265 2024-09-18T16:17:31.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:31.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj274 testdata.562265 2024-09-18T16:17:31.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:31.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj275 testdata.562265 2024-09-18T16:17:31.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:31.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj276 testdata.562265 2024-09-18T16:17:32.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:32.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj277 testdata.562265 2024-09-18T16:17:32.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:32.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj278 testdata.562265 2024-09-18T16:17:32.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:32.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj279 testdata.562265 2024-09-18T16:17:32.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:32.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj280 testdata.562265 2024-09-18T16:17:32.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:32.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj281 testdata.562265 2024-09-18T16:17:32.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:32.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj282 testdata.562265 2024-09-18T16:17:32.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:32.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj283 testdata.562265 2024-09-18T16:17:33.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:33.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj284 testdata.562265 2024-09-18T16:17:33.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:33.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj285 testdata.562265 2024-09-18T16:17:33.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:33.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj286 testdata.562265 2024-09-18T16:17:33.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:33.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj287 testdata.562265 2024-09-18T16:17:33.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:33.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj288 testdata.562265 2024-09-18T16:17:33.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:33.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj289 testdata.562265 2024-09-18T16:17:33.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:33.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj290 testdata.562265 2024-09-18T16:17:34.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:34.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj291 testdata.562265 2024-09-18T16:17:34.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:34.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj292 testdata.562265 2024-09-18T16:17:34.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:34.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj293 testdata.562265 2024-09-18T16:17:34.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:34.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj294 testdata.562265 2024-09-18T16:17:34.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:34.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj295 testdata.562265 2024-09-18T16:17:34.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:34.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj296 testdata.562265 2024-09-18T16:17:34.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:34.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj297 testdata.562265 2024-09-18T16:17:35.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:35.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj298 testdata.562265 2024-09-18T16:17:35.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:35.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj299 testdata.562265 2024-09-18T16:17:35.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:35.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj300 testdata.562265 2024-09-18T16:17:35.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:35.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj301 testdata.562265 2024-09-18T16:17:35.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:35.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj302 testdata.562265 2024-09-18T16:17:35.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:35.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj303 testdata.562265 2024-09-18T16:17:35.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:35.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj304 testdata.562265 2024-09-18T16:17:36.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:36.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj305 testdata.562265 2024-09-18T16:17:36.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:36.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj306 testdata.562265 2024-09-18T16:17:36.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:36.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj307 testdata.562265 2024-09-18T16:17:36.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:36.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj308 testdata.562265 2024-09-18T16:17:36.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:36.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj309 testdata.562265 2024-09-18T16:17:36.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:36.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj310 testdata.562265 2024-09-18T16:17:36.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:36.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj311 testdata.562265 2024-09-18T16:17:37.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:37.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj312 testdata.562265 2024-09-18T16:17:37.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:37.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj313 testdata.562265 2024-09-18T16:17:37.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:37.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj314 testdata.562265 2024-09-18T16:17:37.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:37.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj315 testdata.562265 2024-09-18T16:17:37.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:37.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj316 testdata.562265 2024-09-18T16:17:37.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:37.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj317 testdata.562265 2024-09-18T16:17:37.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:37.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj318 testdata.562265 2024-09-18T16:17:38.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:38.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj319 testdata.562265 2024-09-18T16:17:38.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:38.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj320 testdata.562265 2024-09-18T16:17:38.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:38.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj321 testdata.562265 2024-09-18T16:17:38.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:38.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj322 testdata.562265 2024-09-18T16:17:38.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:38.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj323 testdata.562265 2024-09-18T16:17:38.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:38.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj324 testdata.562265 2024-09-18T16:17:39.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:39.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj325 testdata.562265 2024-09-18T16:17:39.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:39.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj326 testdata.562265 2024-09-18T16:17:39.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:39.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj327 testdata.562265 2024-09-18T16:17:39.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:39.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj328 testdata.562265 2024-09-18T16:17:39.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:39.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj329 testdata.562265 2024-09-18T16:17:39.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:39.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj330 testdata.562265 2024-09-18T16:17:39.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:39.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj331 testdata.562265 2024-09-18T16:17:40.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:40.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj332 testdata.562265 2024-09-18T16:17:40.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:40.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj333 testdata.562265 2024-09-18T16:17:40.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:40.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj334 testdata.562265 2024-09-18T16:17:40.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:40.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj335 testdata.562265 2024-09-18T16:17:40.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:40.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj336 testdata.562265 2024-09-18T16:17:40.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:40.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj337 testdata.562265 2024-09-18T16:17:41.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:41.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj338 testdata.562265 2024-09-18T16:17:41.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:41.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj339 testdata.562265 2024-09-18T16:17:41.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:41.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj340 testdata.562265 2024-09-18T16:17:41.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:41.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj341 testdata.562265 2024-09-18T16:17:41.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:41.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj342 testdata.562265 2024-09-18T16:17:41.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:41.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj343 testdata.562265 2024-09-18T16:17:41.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:41.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj344 testdata.562265 2024-09-18T16:17:42.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:42.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj345 testdata.562265 2024-09-18T16:17:42.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:42.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj346 testdata.562265 2024-09-18T16:17:42.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:42.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj347 testdata.562265 2024-09-18T16:17:42.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:42.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj348 testdata.562265 2024-09-18T16:17:42.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:42.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj349 testdata.562265 2024-09-18T16:17:42.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:42.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj350 testdata.562265 2024-09-18T16:17:42.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:42.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj351 testdata.562265 2024-09-18T16:17:43.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:43.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj352 testdata.562265 2024-09-18T16:17:43.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:43.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj353 testdata.562265 2024-09-18T16:17:43.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:43.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj354 testdata.562265 2024-09-18T16:17:43.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:43.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj355 testdata.562265 2024-09-18T16:17:43.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:43.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj356 testdata.562265 2024-09-18T16:17:43.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:43.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj357 testdata.562265 2024-09-18T16:17:43.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:43.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj358 testdata.562265 2024-09-18T16:17:44.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:44.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj359 testdata.562265 2024-09-18T16:17:44.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:44.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj360 testdata.562265 2024-09-18T16:17:44.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:44.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj361 testdata.562265 2024-09-18T16:17:44.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:44.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj362 testdata.562265 2024-09-18T16:17:44.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:44.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj363 testdata.562265 2024-09-18T16:17:44.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:44.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj364 testdata.562265 2024-09-18T16:17:44.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:44.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj365 testdata.562265 2024-09-18T16:17:45.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:45.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj366 testdata.562265 2024-09-18T16:17:45.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:45.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj367 testdata.562265 2024-09-18T16:17:45.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:45.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj368 testdata.562265 2024-09-18T16:17:45.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:45.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj369 testdata.562265 2024-09-18T16:17:45.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:45.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj370 testdata.562265 2024-09-18T16:17:45.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:45.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj371 testdata.562265 2024-09-18T16:17:45.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:45.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj372 testdata.562265 2024-09-18T16:17:46.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:46.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj373 testdata.562265 2024-09-18T16:17:46.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:46.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj374 testdata.562265 2024-09-18T16:17:46.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:46.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj375 testdata.562265 2024-09-18T16:17:46.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:46.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj376 testdata.562265 2024-09-18T16:17:46.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:46.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj377 testdata.562265 2024-09-18T16:17:46.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:46.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj378 testdata.562265 2024-09-18T16:17:47.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:47.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj379 testdata.562265 2024-09-18T16:17:47.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:47.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj380 testdata.562265 2024-09-18T16:17:47.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:47.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj381 testdata.562265 2024-09-18T16:17:47.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:47.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj382 testdata.562265 2024-09-18T16:17:47.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:47.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj383 testdata.562265 2024-09-18T16:17:47.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:47.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj384 testdata.562265 2024-09-18T16:17:47.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:47.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj385 testdata.562265 2024-09-18T16:17:48.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:48.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj386 testdata.562265 2024-09-18T16:17:48.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:48.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj387 testdata.562265 2024-09-18T16:17:48.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:48.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj388 testdata.562265 2024-09-18T16:17:48.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:48.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj389 testdata.562265 2024-09-18T16:17:48.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:48.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj390 testdata.562265 2024-09-18T16:17:48.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:48.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj391 testdata.562265 2024-09-18T16:17:48.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:48.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj392 testdata.562265 2024-09-18T16:17:49.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:49.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj393 testdata.562265 2024-09-18T16:17:49.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:49.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj394 testdata.562265 2024-09-18T16:17:49.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:49.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj395 testdata.562265 2024-09-18T16:17:49.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:49.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj396 testdata.562265 2024-09-18T16:17:49.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:49.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj397 testdata.562265 2024-09-18T16:17:49.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:49.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj398 testdata.562265 2024-09-18T16:17:49.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:49.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj399 testdata.562265 2024-09-18T16:17:50.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:50.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj400 testdata.562265 2024-09-18T16:17:50.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:50.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj401 testdata.562265 2024-09-18T16:17:50.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:50.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj402 testdata.562265 2024-09-18T16:17:50.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:50.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj403 testdata.562265 2024-09-18T16:17:50.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:50.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj404 testdata.562265 2024-09-18T16:17:50.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:50.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj405 testdata.562265 2024-09-18T16:17:50.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:50.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj406 testdata.562265 2024-09-18T16:17:51.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:51.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj407 testdata.562265 2024-09-18T16:17:51.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:51.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj408 testdata.562265 2024-09-18T16:17:51.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:51.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj409 testdata.562265 2024-09-18T16:17:51.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:51.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj410 testdata.562265 2024-09-18T16:17:51.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:51.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj411 testdata.562265 2024-09-18T16:17:51.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:51.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj412 testdata.562265 2024-09-18T16:17:52.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:52.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj413 testdata.562265 2024-09-18T16:17:52.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:52.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj414 testdata.562265 2024-09-18T16:17:52.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:52.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj415 testdata.562265 2024-09-18T16:17:52.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:52.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj416 testdata.562265 2024-09-18T16:17:52.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:52.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj417 testdata.562265 2024-09-18T16:17:52.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:52.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj418 testdata.562265 2024-09-18T16:17:52.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:52.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj419 testdata.562265 2024-09-18T16:17:53.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:53.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj420 testdata.562265 2024-09-18T16:17:53.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:53.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj421 testdata.562265 2024-09-18T16:17:53.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:53.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj422 testdata.562265 2024-09-18T16:17:53.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:53.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj423 testdata.562265 2024-09-18T16:17:53.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:53.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj424 testdata.562265 2024-09-18T16:17:53.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:53.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj425 testdata.562265 2024-09-18T16:17:53.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:53.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj426 testdata.562265 2024-09-18T16:17:54.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:54.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj427 testdata.562265 2024-09-18T16:17:54.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:54.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj428 testdata.562265 2024-09-18T16:17:54.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:54.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj429 testdata.562265 2024-09-18T16:17:54.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:54.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj430 testdata.562265 2024-09-18T16:17:54.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:54.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj431 testdata.562265 2024-09-18T16:17:54.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:54.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj432 testdata.562265 2024-09-18T16:17:55.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:55.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj433 testdata.562265 2024-09-18T16:17:55.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:55.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj434 testdata.562265 2024-09-18T16:17:55.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:55.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj435 testdata.562265 2024-09-18T16:17:55.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:55.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj436 testdata.562265 2024-09-18T16:17:55.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:55.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj437 testdata.562265 2024-09-18T16:17:55.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:55.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj438 testdata.562265 2024-09-18T16:17:55.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:55.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj439 testdata.562265 2024-09-18T16:17:56.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:56.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj440 testdata.562265 2024-09-18T16:17:56.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:56.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj441 testdata.562265 2024-09-18T16:17:56.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:56.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj442 testdata.562265 2024-09-18T16:17:56.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:56.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj443 testdata.562265 2024-09-18T16:17:56.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:56.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj444 testdata.562265 2024-09-18T16:17:56.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:56.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj445 testdata.562265 2024-09-18T16:17:56.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:56.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj446 testdata.562265 2024-09-18T16:17:57.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:57.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj447 testdata.562265 2024-09-18T16:17:57.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:57.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj448 testdata.562265 2024-09-18T16:17:57.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:57.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj449 testdata.562265 2024-09-18T16:17:57.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:57.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj450 testdata.562265 2024-09-18T16:17:57.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:57.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj451 testdata.562265 2024-09-18T16:17:57.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:57.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj452 testdata.562265 2024-09-18T16:17:58.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:58.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj453 testdata.562265 2024-09-18T16:17:58.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:58.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj454 testdata.562265 2024-09-18T16:17:58.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:58.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj455 testdata.562265 2024-09-18T16:17:58.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:58.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj456 testdata.562265 2024-09-18T16:17:58.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:58.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj457 testdata.562265 2024-09-18T16:17:58.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:58.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj458 testdata.562265 2024-09-18T16:17:58.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:58.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj459 testdata.562265 2024-09-18T16:17:59.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:59.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj460 testdata.562265 2024-09-18T16:17:59.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:59.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj461 testdata.562265 2024-09-18T16:17:59.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:59.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj462 testdata.562265 2024-09-18T16:17:59.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:59.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj463 testdata.562265 2024-09-18T16:17:59.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:59.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj464 testdata.562265 2024-09-18T16:17:59.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:59.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj465 testdata.562265 2024-09-18T16:17:59.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:17:59.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj466 testdata.562265 2024-09-18T16:18:00.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:00.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj467 testdata.562265 2024-09-18T16:18:00.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:00.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj468 testdata.562265 2024-09-18T16:18:00.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:00.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj469 testdata.562265 2024-09-18T16:18:00.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:00.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj470 testdata.562265 2024-09-18T16:18:00.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:00.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj471 testdata.562265 2024-09-18T16:18:00.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:00.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj472 testdata.562265 2024-09-18T16:18:00.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:00.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj473 testdata.562265 2024-09-18T16:18:01.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:01.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj474 testdata.562265 2024-09-18T16:18:01.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:01.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj475 testdata.562265 2024-09-18T16:18:01.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:01.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj476 testdata.562265 2024-09-18T16:18:01.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:01.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj477 testdata.562265 2024-09-18T16:18:01.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:01.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj478 testdata.562265 2024-09-18T16:18:01.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:01.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj479 testdata.562265 2024-09-18T16:18:02.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:02.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj480 testdata.562265 2024-09-18T16:18:02.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:02.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj481 testdata.562265 2024-09-18T16:18:02.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:02.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj482 testdata.562265 2024-09-18T16:18:02.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:02.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj483 testdata.562265 2024-09-18T16:18:02.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:02.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj484 testdata.562265 2024-09-18T16:18:03.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:03.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj485 testdata.562265 2024-09-18T16:18:03.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:03.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj486 testdata.562265 2024-09-18T16:18:03.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:03.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj487 testdata.562265 2024-09-18T16:18:03.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:03.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj488 testdata.562265 2024-09-18T16:18:03.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:03.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj489 testdata.562265 2024-09-18T16:18:03.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:03.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj490 testdata.562265 2024-09-18T16:18:03.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:03.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj491 testdata.562265 2024-09-18T16:18:04.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:04.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj492 testdata.562265 2024-09-18T16:18:04.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:04.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj493 testdata.562265 2024-09-18T16:18:04.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:04.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj494 testdata.562265 2024-09-18T16:18:04.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:04.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj495 testdata.562265 2024-09-18T16:18:04.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:04.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj496 testdata.562265 2024-09-18T16:18:04.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:04.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj497 testdata.562265 2024-09-18T16:18:05.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:05.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj498 testdata.562265 2024-09-18T16:18:05.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:05.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj499 testdata.562265 2024-09-18T16:18:05.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:05.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj500 testdata.562265 2024-09-18T16:18:05.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:05.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj501 testdata.562265 2024-09-18T16:18:05.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:05.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj502 testdata.562265 2024-09-18T16:18:05.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:05.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj503 testdata.562265 2024-09-18T16:18:06.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:06.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj504 testdata.562265 2024-09-18T16:18:06.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:06.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj505 testdata.562265 2024-09-18T16:18:06.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:06.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj506 testdata.562265 2024-09-18T16:18:06.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:06.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj507 testdata.562265 2024-09-18T16:18:06.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:06.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj508 testdata.562265 2024-09-18T16:18:06.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:06.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj509 testdata.562265 2024-09-18T16:18:06.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:06.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj510 testdata.562265 2024-09-18T16:18:06.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:06.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj511 testdata.562265 2024-09-18T16:18:07.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:07.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj512 testdata.562265 2024-09-18T16:18:07.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:07.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj513 testdata.562265 2024-09-18T16:18:07.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:07.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj514 testdata.562265 2024-09-18T16:18:07.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:07.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj515 testdata.562265 2024-09-18T16:18:07.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:07.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj516 testdata.562265 2024-09-18T16:18:07.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:07.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj517 testdata.562265 2024-09-18T16:18:08.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:08.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj518 testdata.562265 2024-09-18T16:18:08.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:08.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj519 testdata.562265 2024-09-18T16:18:08.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:08.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj520 testdata.562265 2024-09-18T16:18:08.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:08.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj521 testdata.562265 2024-09-18T16:18:08.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:08.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj522 testdata.562265 2024-09-18T16:18:08.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:08.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj523 testdata.562265 2024-09-18T16:18:08.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:08.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj524 testdata.562265 2024-09-18T16:18:09.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:09.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj525 testdata.562265 2024-09-18T16:18:09.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:09.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj526 testdata.562265 2024-09-18T16:18:09.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:09.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj527 testdata.562265 2024-09-18T16:18:09.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:09.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj528 testdata.562265 2024-09-18T16:18:09.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:09.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj529 testdata.562265 2024-09-18T16:18:09.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:09.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj530 testdata.562265 2024-09-18T16:18:09.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:09.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj531 testdata.562265 2024-09-18T16:18:10.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:10.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj532 testdata.562265 2024-09-18T16:18:10.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:10.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj533 testdata.562265 2024-09-18T16:18:10.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:10.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj534 testdata.562265 2024-09-18T16:18:10.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:10.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj535 testdata.562265 2024-09-18T16:18:10.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:10.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj536 testdata.562265 2024-09-18T16:18:10.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:10.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj537 testdata.562265 2024-09-18T16:18:10.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:10.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj538 testdata.562265 2024-09-18T16:18:10.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:10.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj539 testdata.562265 2024-09-18T16:18:11.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:11.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj540 testdata.562265 2024-09-18T16:18:11.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:11.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj541 testdata.562265 2024-09-18T16:18:11.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:11.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj542 testdata.562265 2024-09-18T16:18:11.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:11.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj543 testdata.562265 2024-09-18T16:18:11.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:11.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj544 testdata.562265 2024-09-18T16:18:11.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:11.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj545 testdata.562265 2024-09-18T16:18:11.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:11.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj546 testdata.562265 2024-09-18T16:18:12.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:12.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj547 testdata.562265 2024-09-18T16:18:12.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:12.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj548 testdata.562265 2024-09-18T16:18:12.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:12.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj549 testdata.562265 2024-09-18T16:18:12.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:12.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj550 testdata.562265 2024-09-18T16:18:12.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:12.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj551 testdata.562265 2024-09-18T16:18:12.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:12.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj552 testdata.562265 2024-09-18T16:18:13.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:13.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj553 testdata.562265 2024-09-18T16:18:13.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:13.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj554 testdata.562265 2024-09-18T16:18:13.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:13.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj555 testdata.562265 2024-09-18T16:18:13.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:13.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj556 testdata.562265 2024-09-18T16:18:13.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:13.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj557 testdata.562265 2024-09-18T16:18:13.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:13.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj558 testdata.562265 2024-09-18T16:18:13.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:13.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj559 testdata.562265 2024-09-18T16:18:14.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:14.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj560 testdata.562265 2024-09-18T16:18:14.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:14.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj561 testdata.562265 2024-09-18T16:18:14.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:14.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj562 testdata.562265 2024-09-18T16:18:14.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:14.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj563 testdata.562265 2024-09-18T16:18:14.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:14.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj564 testdata.562265 2024-09-18T16:18:14.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:14.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj565 testdata.562265 2024-09-18T16:18:14.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:14.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj566 testdata.562265 2024-09-18T16:18:15.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:15.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj567 testdata.562265 2024-09-18T16:18:15.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:15.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj568 testdata.562265 2024-09-18T16:18:15.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:15.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj569 testdata.562265 2024-09-18T16:18:15.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:15.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj570 testdata.562265 2024-09-18T16:18:15.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:15.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj571 testdata.562265 2024-09-18T16:18:15.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:15.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj572 testdata.562265 2024-09-18T16:18:16.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:16.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj573 testdata.562265 2024-09-18T16:18:16.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:16.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj574 testdata.562265 2024-09-18T16:18:16.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:16.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj575 testdata.562265 2024-09-18T16:18:16.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:16.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj576 testdata.562265 2024-09-18T16:18:16.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:16.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj577 testdata.562265 2024-09-18T16:18:16.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:16.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj578 testdata.562265 2024-09-18T16:18:17.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:17.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj579 testdata.562265 2024-09-18T16:18:17.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:17.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj580 testdata.562265 2024-09-18T16:18:17.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:17.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj581 testdata.562265 2024-09-18T16:18:17.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:17.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj582 testdata.562265 2024-09-18T16:18:17.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:17.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj583 testdata.562265 2024-09-18T16:18:17.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:17.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj584 testdata.562265 2024-09-18T16:18:18.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:18.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj585 testdata.562265 2024-09-18T16:18:18.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:18.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj586 testdata.562265 2024-09-18T16:18:18.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:18.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj587 testdata.562265 2024-09-18T16:18:18.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:18.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj588 testdata.562265 2024-09-18T16:18:18.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:18.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj589 testdata.562265 2024-09-18T16:18:18.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:18.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj590 testdata.562265 2024-09-18T16:18:18.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:18.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj591 testdata.562265 2024-09-18T16:18:19.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:19.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj592 testdata.562265 2024-09-18T16:18:19.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:19.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj593 testdata.562265 2024-09-18T16:18:19.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:19.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj594 testdata.562265 2024-09-18T16:18:19.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:19.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj595 testdata.562265 2024-09-18T16:18:19.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:19.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj596 testdata.562265 2024-09-18T16:18:19.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:19.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj597 testdata.562265 2024-09-18T16:18:19.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:19.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj598 testdata.562265 2024-09-18T16:18:20.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:20.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj599 testdata.562265 2024-09-18T16:18:20.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:20.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj600 testdata.562265 2024-09-18T16:18:20.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:20.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj601 testdata.562265 2024-09-18T16:18:20.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:20.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj602 testdata.562265 2024-09-18T16:18:20.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:20.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj603 testdata.562265 2024-09-18T16:18:20.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:20.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj604 testdata.562265 2024-09-18T16:18:21.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:21.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj605 testdata.562265 2024-09-18T16:18:21.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:21.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj606 testdata.562265 2024-09-18T16:18:21.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:21.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj607 testdata.562265 2024-09-18T16:18:21.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:21.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj608 testdata.562265 2024-09-18T16:18:21.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:21.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj609 testdata.562265 2024-09-18T16:18:21.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:21.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj610 testdata.562265 2024-09-18T16:18:22.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:22.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj611 testdata.562265 2024-09-18T16:18:22.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:22.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj612 testdata.562265 2024-09-18T16:18:22.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:22.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj613 testdata.562265 2024-09-18T16:18:22.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:22.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj614 testdata.562265 2024-09-18T16:18:22.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:22.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj615 testdata.562265 2024-09-18T16:18:22.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:22.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj616 testdata.562265 2024-09-18T16:18:23.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:23.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj617 testdata.562265 2024-09-18T16:18:23.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:23.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj618 testdata.562265 2024-09-18T16:18:23.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:23.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj619 testdata.562265 2024-09-18T16:18:23.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:23.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj620 testdata.562265 2024-09-18T16:18:23.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:23.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj621 testdata.562265 2024-09-18T16:18:23.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:23.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj622 testdata.562265 2024-09-18T16:18:23.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:23.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj623 testdata.562265 2024-09-18T16:18:24.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:24.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj624 testdata.562265 2024-09-18T16:18:24.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:24.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj625 testdata.562265 2024-09-18T16:18:24.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:24.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj626 testdata.562265 2024-09-18T16:18:24.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:24.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj627 testdata.562265 2024-09-18T16:18:24.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:24.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj628 testdata.562265 2024-09-18T16:18:24.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:24.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj629 testdata.562265 2024-09-18T16:18:24.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:24.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj630 testdata.562265 2024-09-18T16:18:25.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:25.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj631 testdata.562265 2024-09-18T16:18:25.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:25.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj632 testdata.562265 2024-09-18T16:18:25.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:25.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj633 testdata.562265 2024-09-18T16:18:25.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:25.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj634 testdata.562265 2024-09-18T16:18:25.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:25.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj635 testdata.562265 2024-09-18T16:18:25.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:25.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj636 testdata.562265 2024-09-18T16:18:26.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:26.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj637 testdata.562265 2024-09-18T16:18:26.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:26.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj638 testdata.562265 2024-09-18T16:18:26.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:26.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj639 testdata.562265 2024-09-18T16:18:26.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:26.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj640 testdata.562265 2024-09-18T16:18:26.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:26.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj641 testdata.562265 2024-09-18T16:18:26.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:26.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj642 testdata.562265 2024-09-18T16:18:26.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:26.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj643 testdata.562265 2024-09-18T16:18:27.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:27.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj644 testdata.562265 2024-09-18T16:18:27.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:27.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj645 testdata.562265 2024-09-18T16:18:27.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:27.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj646 testdata.562265 2024-09-18T16:18:27.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:27.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj647 testdata.562265 2024-09-18T16:18:27.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:27.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj648 testdata.562265 2024-09-18T16:18:27.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:27.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj649 testdata.562265 2024-09-18T16:18:28.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:28.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj650 testdata.562265 2024-09-18T16:18:28.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:28.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj651 testdata.562265 2024-09-18T16:18:28.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:28.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj652 testdata.562265 2024-09-18T16:18:28.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:28.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj653 testdata.562265 2024-09-18T16:18:28.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:28.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj654 testdata.562265 2024-09-18T16:18:28.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:28.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj655 testdata.562265 2024-09-18T16:18:28.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:28.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj656 testdata.562265 2024-09-18T16:18:29.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:29.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj657 testdata.562265 2024-09-18T16:18:29.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:29.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj658 testdata.562265 2024-09-18T16:18:29.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:29.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj659 testdata.562265 2024-09-18T16:18:29.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:29.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj660 testdata.562265 2024-09-18T16:18:29.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:29.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj661 testdata.562265 2024-09-18T16:18:29.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:29.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj662 testdata.562265 2024-09-18T16:18:29.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:29.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj663 testdata.562265 2024-09-18T16:18:30.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:30.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj664 testdata.562265 2024-09-18T16:18:30.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:30.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj665 testdata.562265 2024-09-18T16:18:30.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:30.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj666 testdata.562265 2024-09-18T16:18:30.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:30.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj667 testdata.562265 2024-09-18T16:18:30.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:30.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj668 testdata.562265 2024-09-18T16:18:30.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:30.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj669 testdata.562265 2024-09-18T16:18:31.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:31.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj670 testdata.562265 2024-09-18T16:18:31.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:31.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj671 testdata.562265 2024-09-18T16:18:31.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:31.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj672 testdata.562265 2024-09-18T16:18:31.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:31.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj673 testdata.562265 2024-09-18T16:18:31.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:31.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj674 testdata.562265 2024-09-18T16:18:31.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:31.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj675 testdata.562265 2024-09-18T16:18:31.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:31.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj676 testdata.562265 2024-09-18T16:18:32.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:32.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj677 testdata.562265 2024-09-18T16:18:32.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:32.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj678 testdata.562265 2024-09-18T16:18:32.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:32.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj679 testdata.562265 2024-09-18T16:18:32.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:32.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj680 testdata.562265 2024-09-18T16:18:32.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:32.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj681 testdata.562265 2024-09-18T16:18:32.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:32.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj682 testdata.562265 2024-09-18T16:18:32.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:32.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj683 testdata.562265 2024-09-18T16:18:33.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:33.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj684 testdata.562265 2024-09-18T16:18:33.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:33.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj685 testdata.562265 2024-09-18T16:18:33.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:33.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj686 testdata.562265 2024-09-18T16:18:34.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:34.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj687 testdata.562265 2024-09-18T16:18:34.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:34.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj688 testdata.562265 2024-09-18T16:18:34.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:34.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj689 testdata.562265 2024-09-18T16:18:34.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:34.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj690 testdata.562265 2024-09-18T16:18:34.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:34.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj691 testdata.562265 2024-09-18T16:18:34.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:34.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj692 testdata.562265 2024-09-18T16:18:34.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:34.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj693 testdata.562265 2024-09-18T16:18:35.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:35.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj694 testdata.562265 2024-09-18T16:18:35.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:35.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj695 testdata.562265 2024-09-18T16:18:35.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:35.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj696 testdata.562265 2024-09-18T16:18:35.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:35.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj697 testdata.562265 2024-09-18T16:18:35.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:35.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj698 testdata.562265 2024-09-18T16:18:35.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:35.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj699 testdata.562265 2024-09-18T16:18:35.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:35.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj700 testdata.562265 2024-09-18T16:18:36.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:36.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj701 testdata.562265 2024-09-18T16:18:36.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:36.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj702 testdata.562265 2024-09-18T16:18:36.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:36.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj703 testdata.562265 2024-09-18T16:18:36.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:36.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj704 testdata.562265 2024-09-18T16:18:36.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:36.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj705 testdata.562265 2024-09-18T16:18:36.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:36.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj706 testdata.562265 2024-09-18T16:18:36.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:36.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj707 testdata.562265 2024-09-18T16:18:37.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:37.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj708 testdata.562265 2024-09-18T16:18:37.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:37.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj709 testdata.562265 2024-09-18T16:18:37.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:37.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj710 testdata.562265 2024-09-18T16:18:37.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:37.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj711 testdata.562265 2024-09-18T16:18:37.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:37.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj712 testdata.562265 2024-09-18T16:18:37.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:37.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj713 testdata.562265 2024-09-18T16:18:38.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:38.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj714 testdata.562265 2024-09-18T16:18:38.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:38.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj715 testdata.562265 2024-09-18T16:18:38.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:38.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj716 testdata.562265 2024-09-18T16:18:38.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:38.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj717 testdata.562265 2024-09-18T16:18:38.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:38.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj718 testdata.562265 2024-09-18T16:18:38.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:38.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj719 testdata.562265 2024-09-18T16:18:38.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:38.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj720 testdata.562265 2024-09-18T16:18:39.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:39.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj721 testdata.562265 2024-09-18T16:18:39.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:39.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj722 testdata.562265 2024-09-18T16:18:39.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:39.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj723 testdata.562265 2024-09-18T16:18:39.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:39.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj724 testdata.562265 2024-09-18T16:18:39.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:39.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj725 testdata.562265 2024-09-18T16:18:39.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:39.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj726 testdata.562265 2024-09-18T16:18:39.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:39.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj727 testdata.562265 2024-09-18T16:18:40.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:40.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj728 testdata.562265 2024-09-18T16:18:40.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:40.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj729 testdata.562265 2024-09-18T16:18:40.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:40.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj730 testdata.562265 2024-09-18T16:18:40.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:40.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj731 testdata.562265 2024-09-18T16:18:40.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:40.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj732 testdata.562265 2024-09-18T16:18:40.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:40.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj733 testdata.562265 2024-09-18T16:18:40.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:40.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj734 testdata.562265 2024-09-18T16:18:41.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:41.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj735 testdata.562265 2024-09-18T16:18:41.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:41.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj736 testdata.562265 2024-09-18T16:18:41.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:41.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj737 testdata.562265 2024-09-18T16:18:41.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:41.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj738 testdata.562265 2024-09-18T16:18:41.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:41.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj739 testdata.562265 2024-09-18T16:18:41.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:41.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj740 testdata.562265 2024-09-18T16:18:42.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:42.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj741 testdata.562265 2024-09-18T16:18:42.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:42.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj742 testdata.562265 2024-09-18T16:18:42.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:42.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj743 testdata.562265 2024-09-18T16:18:42.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:42.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj744 testdata.562265 2024-09-18T16:18:42.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:42.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj745 testdata.562265 2024-09-18T16:18:42.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:42.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj746 testdata.562265 2024-09-18T16:18:42.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:42.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj747 testdata.562265 2024-09-18T16:18:43.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:43.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj748 testdata.562265 2024-09-18T16:18:43.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:43.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj749 testdata.562265 2024-09-18T16:18:43.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:43.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj750 testdata.562265 2024-09-18T16:18:43.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:43.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj751 testdata.562265 2024-09-18T16:18:43.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:43.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj752 testdata.562265 2024-09-18T16:18:43.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:43.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj753 testdata.562265 2024-09-18T16:18:43.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:43.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj754 testdata.562265 2024-09-18T16:18:44.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:44.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj755 testdata.562265 2024-09-18T16:18:44.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:44.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj756 testdata.562265 2024-09-18T16:18:44.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:44.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj757 testdata.562265 2024-09-18T16:18:44.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:44.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj758 testdata.562265 2024-09-18T16:18:44.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:44.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj759 testdata.562265 2024-09-18T16:18:44.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:44.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj760 testdata.562265 2024-09-18T16:18:45.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:45.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj761 testdata.562265 2024-09-18T16:18:45.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:45.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj762 testdata.562265 2024-09-18T16:18:45.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:45.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj763 testdata.562265 2024-09-18T16:18:45.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:45.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj764 testdata.562265 2024-09-18T16:18:45.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:45.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj765 testdata.562265 2024-09-18T16:18:45.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:45.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj766 testdata.562265 2024-09-18T16:18:45.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:45.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj767 testdata.562265 2024-09-18T16:18:46.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:46.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj768 testdata.562265 2024-09-18T16:18:46.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:46.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj769 testdata.562265 2024-09-18T16:18:46.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:46.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj770 testdata.562265 2024-09-18T16:18:46.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:46.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj771 testdata.562265 2024-09-18T16:18:46.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:46.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj772 testdata.562265 2024-09-18T16:18:46.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:46.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj773 testdata.562265 2024-09-18T16:18:47.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:47.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj774 testdata.562265 2024-09-18T16:18:47.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:47.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj775 testdata.562265 2024-09-18T16:18:47.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:47.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj776 testdata.562265 2024-09-18T16:18:47.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:47.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj777 testdata.562265 2024-09-18T16:18:47.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:47.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj778 testdata.562265 2024-09-18T16:18:47.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:47.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj779 testdata.562265 2024-09-18T16:18:47.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:47.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj780 testdata.562265 2024-09-18T16:18:48.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:48.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj781 testdata.562265 2024-09-18T16:18:48.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:48.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj782 testdata.562265 2024-09-18T16:18:48.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:48.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj783 testdata.562265 2024-09-18T16:18:48.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:48.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj784 testdata.562265 2024-09-18T16:18:48.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:48.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj785 testdata.562265 2024-09-18T16:18:48.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:48.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj786 testdata.562265 2024-09-18T16:18:48.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:48.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj787 testdata.562265 2024-09-18T16:18:48.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:48.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj788 testdata.562265 2024-09-18T16:18:49.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:49.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj789 testdata.562265 2024-09-18T16:18:49.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:49.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj790 testdata.562265 2024-09-18T16:18:49.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:49.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj791 testdata.562265 2024-09-18T16:18:49.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:49.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj792 testdata.562265 2024-09-18T16:18:49.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:49.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj793 testdata.562265 2024-09-18T16:18:49.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:49.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj794 testdata.562265 2024-09-18T16:18:50.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:50.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj795 testdata.562265 2024-09-18T16:18:50.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:50.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj796 testdata.562265 2024-09-18T16:18:50.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:50.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj797 testdata.562265 2024-09-18T16:18:50.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:50.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj798 testdata.562265 2024-09-18T16:18:50.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:50.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj799 testdata.562265 2024-09-18T16:18:50.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:50.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj800 testdata.562265 2024-09-18T16:18:50.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:50.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj801 testdata.562265 2024-09-18T16:18:51.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:51.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj802 testdata.562265 2024-09-18T16:18:51.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:51.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj803 testdata.562265 2024-09-18T16:18:51.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:51.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj804 testdata.562265 2024-09-18T16:18:51.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:51.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj805 testdata.562265 2024-09-18T16:18:51.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:51.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj806 testdata.562265 2024-09-18T16:18:51.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:51.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj807 testdata.562265 2024-09-18T16:18:52.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:52.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj808 testdata.562265 2024-09-18T16:18:52.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:52.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj809 testdata.562265 2024-09-18T16:18:52.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:52.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj810 testdata.562265 2024-09-18T16:18:52.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:52.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj811 testdata.562265 2024-09-18T16:18:52.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:52.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj812 testdata.562265 2024-09-18T16:18:52.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:52.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj813 testdata.562265 2024-09-18T16:18:52.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:52.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj814 testdata.562265 2024-09-18T16:18:53.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:53.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj815 testdata.562265 2024-09-18T16:18:53.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:53.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj816 testdata.562265 2024-09-18T16:18:53.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:53.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj817 testdata.562265 2024-09-18T16:18:53.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:53.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj818 testdata.562265 2024-09-18T16:18:53.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:53.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj819 testdata.562265 2024-09-18T16:18:53.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:53.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj820 testdata.562265 2024-09-18T16:18:53.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:53.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj821 testdata.562265 2024-09-18T16:18:54.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:54.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj822 testdata.562265 2024-09-18T16:18:54.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:54.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj823 testdata.562265 2024-09-18T16:18:54.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:54.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj824 testdata.562265 2024-09-18T16:18:54.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:54.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj825 testdata.562265 2024-09-18T16:18:54.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:54.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj826 testdata.562265 2024-09-18T16:18:54.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:54.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj827 testdata.562265 2024-09-18T16:18:55.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:55.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj828 testdata.562265 2024-09-18T16:18:55.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:55.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj829 testdata.562265 2024-09-18T16:18:55.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:55.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj830 testdata.562265 2024-09-18T16:18:55.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:55.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj831 testdata.562265 2024-09-18T16:18:55.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:55.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj832 testdata.562265 2024-09-18T16:18:55.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:55.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj833 testdata.562265 2024-09-18T16:18:55.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:55.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj834 testdata.562265 2024-09-18T16:18:56.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:56.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj835 testdata.562265 2024-09-18T16:18:56.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:56.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj836 testdata.562265 2024-09-18T16:18:56.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:56.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj837 testdata.562265 2024-09-18T16:18:56.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:56.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj838 testdata.562265 2024-09-18T16:18:56.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:56.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj839 testdata.562265 2024-09-18T16:18:56.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:56.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj840 testdata.562265 2024-09-18T16:18:57.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:57.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj841 testdata.562265 2024-09-18T16:18:57.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:57.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj842 testdata.562265 2024-09-18T16:18:57.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:57.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj843 testdata.562265 2024-09-18T16:18:57.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:57.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj844 testdata.562265 2024-09-18T16:18:57.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:57.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj845 testdata.562265 2024-09-18T16:18:57.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:57.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj846 testdata.562265 2024-09-18T16:18:57.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:57.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj847 testdata.562265 2024-09-18T16:18:58.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:58.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj848 testdata.562265 2024-09-18T16:18:58.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:58.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj849 testdata.562265 2024-09-18T16:18:58.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:58.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj850 testdata.562265 2024-09-18T16:18:58.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:58.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj851 testdata.562265 2024-09-18T16:18:58.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:58.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj852 testdata.562265 2024-09-18T16:18:58.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:58.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj853 testdata.562265 2024-09-18T16:18:58.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:58.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj854 testdata.562265 2024-09-18T16:18:59.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:59.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj855 testdata.562265 2024-09-18T16:18:59.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:59.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj856 testdata.562265 2024-09-18T16:18:59.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:59.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj857 testdata.562265 2024-09-18T16:18:59.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:59.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj858 testdata.562265 2024-09-18T16:18:59.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:59.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj859 testdata.562265 2024-09-18T16:18:59.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:18:59.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj860 testdata.562265 2024-09-18T16:19:00.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:00.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj861 testdata.562265 2024-09-18T16:19:00.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:00.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj862 testdata.562265 2024-09-18T16:19:00.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:00.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj863 testdata.562265 2024-09-18T16:19:00.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:00.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj864 testdata.562265 2024-09-18T16:19:00.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:00.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj865 testdata.562265 2024-09-18T16:19:00.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:00.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj866 testdata.562265 2024-09-18T16:19:01.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:01.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj867 testdata.562265 2024-09-18T16:19:01.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:01.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj868 testdata.562265 2024-09-18T16:19:01.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:01.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj869 testdata.562265 2024-09-18T16:19:01.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:01.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj870 testdata.562265 2024-09-18T16:19:01.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:01.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj871 testdata.562265 2024-09-18T16:19:01.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:01.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj872 testdata.562265 2024-09-18T16:19:01.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:01.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj873 testdata.562265 2024-09-18T16:19:02.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:02.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj874 testdata.562265 2024-09-18T16:19:02.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:02.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj875 testdata.562265 2024-09-18T16:19:02.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:02.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj876 testdata.562265 2024-09-18T16:19:02.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:02.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj877 testdata.562265 2024-09-18T16:19:02.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:02.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj878 testdata.562265 2024-09-18T16:19:02.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:02.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj879 testdata.562265 2024-09-18T16:19:02.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:02.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj880 testdata.562265 2024-09-18T16:19:03.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:03.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj881 testdata.562265 2024-09-18T16:19:03.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:03.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj882 testdata.562265 2024-09-18T16:19:03.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:03.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj883 testdata.562265 2024-09-18T16:19:03.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:03.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj884 testdata.562265 2024-09-18T16:19:03.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:03.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj885 testdata.562265 2024-09-18T16:19:03.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:03.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj886 testdata.562265 2024-09-18T16:19:04.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:04.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj887 testdata.562265 2024-09-18T16:19:04.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:04.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj888 testdata.562265 2024-09-18T16:19:04.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:04.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj889 testdata.562265 2024-09-18T16:19:04.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:04.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj890 testdata.562265 2024-09-18T16:19:04.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:04.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj891 testdata.562265 2024-09-18T16:19:05.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:05.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj892 testdata.562265 2024-09-18T16:19:05.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:05.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj893 testdata.562265 2024-09-18T16:19:05.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:05.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj894 testdata.562265 2024-09-18T16:19:05.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:05.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj895 testdata.562265 2024-09-18T16:19:05.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:05.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj896 testdata.562265 2024-09-18T16:19:05.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:05.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj897 testdata.562265 2024-09-18T16:19:05.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:05.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj898 testdata.562265 2024-09-18T16:19:06.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:06.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj899 testdata.562265 2024-09-18T16:19:06.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:06.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj900 testdata.562265 2024-09-18T16:19:06.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:06.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj901 testdata.562265 2024-09-18T16:19:06.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:06.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj902 testdata.562265 2024-09-18T16:19:06.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:06.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj903 testdata.562265 2024-09-18T16:19:06.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:06.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj904 testdata.562265 2024-09-18T16:19:06.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:06.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj905 testdata.562265 2024-09-18T16:19:07.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:07.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj906 testdata.562265 2024-09-18T16:19:07.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:07.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj907 testdata.562265 2024-09-18T16:19:07.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:07.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj908 testdata.562265 2024-09-18T16:19:07.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:07.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj909 testdata.562265 2024-09-18T16:19:07.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:07.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj910 testdata.562265 2024-09-18T16:19:07.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:07.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj911 testdata.562265 2024-09-18T16:19:07.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:07.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj912 testdata.562265 2024-09-18T16:19:08.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:08.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj913 testdata.562265 2024-09-18T16:19:08.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:08.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj914 testdata.562265 2024-09-18T16:19:08.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:08.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj915 testdata.562265 2024-09-18T16:19:08.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:08.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj916 testdata.562265 2024-09-18T16:19:08.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:08.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj917 testdata.562265 2024-09-18T16:19:08.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:08.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj918 testdata.562265 2024-09-18T16:19:08.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:08.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj919 testdata.562265 2024-09-18T16:19:09.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:09.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj920 testdata.562265 2024-09-18T16:19:09.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:09.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj921 testdata.562265 2024-09-18T16:19:09.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:09.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj922 testdata.562265 2024-09-18T16:19:09.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:09.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj923 testdata.562265 2024-09-18T16:19:09.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:09.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj924 testdata.562265 2024-09-18T16:19:09.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:09.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj925 testdata.562265 2024-09-18T16:19:10.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:10.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj926 testdata.562265 2024-09-18T16:19:10.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:10.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj927 testdata.562265 2024-09-18T16:19:10.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:10.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj928 testdata.562265 2024-09-18T16:19:10.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:10.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj929 testdata.562265 2024-09-18T16:19:10.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:10.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj930 testdata.562265 2024-09-18T16:19:10.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:10.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj931 testdata.562265 2024-09-18T16:19:11.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:11.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj932 testdata.562265 2024-09-18T16:19:11.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:11.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj933 testdata.562265 2024-09-18T16:19:11.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:11.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj934 testdata.562265 2024-09-18T16:19:11.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:11.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj935 testdata.562265 2024-09-18T16:19:11.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:11.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj936 testdata.562265 2024-09-18T16:19:11.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:11.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj937 testdata.562265 2024-09-18T16:19:11.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:11.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj938 testdata.562265 2024-09-18T16:19:12.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:12.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj939 testdata.562265 2024-09-18T16:19:12.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:12.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj940 testdata.562265 2024-09-18T16:19:12.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:12.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj941 testdata.562265 2024-09-18T16:19:12.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:12.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj942 testdata.562265 2024-09-18T16:19:12.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:12.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj943 testdata.562265 2024-09-18T16:19:12.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:12.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj944 testdata.562265 2024-09-18T16:19:12.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:12.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj945 testdata.562265 2024-09-18T16:19:13.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:13.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj946 testdata.562265 2024-09-18T16:19:13.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:13.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj947 testdata.562265 2024-09-18T16:19:13.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:13.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj948 testdata.562265 2024-09-18T16:19:13.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:13.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj949 testdata.562265 2024-09-18T16:19:13.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:13.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj950 testdata.562265 2024-09-18T16:19:13.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:13.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj951 testdata.562265 2024-09-18T16:19:13.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:13.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj952 testdata.562265 2024-09-18T16:19:14.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:14.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj953 testdata.562265 2024-09-18T16:19:14.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:14.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj954 testdata.562265 2024-09-18T16:19:14.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:14.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj955 testdata.562265 2024-09-18T16:19:14.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:14.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj956 testdata.562265 2024-09-18T16:19:14.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:14.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj957 testdata.562265 2024-09-18T16:19:14.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:14.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj958 testdata.562265 2024-09-18T16:19:15.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:15.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj959 testdata.562265 2024-09-18T16:19:15.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:15.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj960 testdata.562265 2024-09-18T16:19:15.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:15.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj961 testdata.562265 2024-09-18T16:19:15.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:15.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj962 testdata.562265 2024-09-18T16:19:15.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:15.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj963 testdata.562265 2024-09-18T16:19:15.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:15.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj964 testdata.562265 2024-09-18T16:19:15.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:15.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj965 testdata.562265 2024-09-18T16:19:16.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:16.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj966 testdata.562265 2024-09-18T16:19:16.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:16.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj967 testdata.562265 2024-09-18T16:19:16.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:16.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj968 testdata.562265 2024-09-18T16:19:16.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:16.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj969 testdata.562265 2024-09-18T16:19:16.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:16.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj970 testdata.562265 2024-09-18T16:19:16.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:16.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj971 testdata.562265 2024-09-18T16:19:17.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:17.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj972 testdata.562265 2024-09-18T16:19:17.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:17.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj973 testdata.562265 2024-09-18T16:19:17.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:17.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj974 testdata.562265 2024-09-18T16:19:17.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:17.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj975 testdata.562265 2024-09-18T16:19:17.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:17.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj976 testdata.562265 2024-09-18T16:19:17.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:17.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj977 testdata.562265 2024-09-18T16:19:17.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:17.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj978 testdata.562265 2024-09-18T16:19:18.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:18.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj979 testdata.562265 2024-09-18T16:19:18.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:18.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj980 testdata.562265 2024-09-18T16:19:18.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:18.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj981 testdata.562265 2024-09-18T16:19:18.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:18.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj982 testdata.562265 2024-09-18T16:19:18.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:18.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj983 testdata.562265 2024-09-18T16:19:18.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:18.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj984 testdata.562265 2024-09-18T16:19:19.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:19.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj985 testdata.562265 2024-09-18T16:19:19.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:19.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj986 testdata.562265 2024-09-18T16:19:19.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:19.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj987 testdata.562265 2024-09-18T16:19:19.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:19.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj988 testdata.562265 2024-09-18T16:19:19.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:19.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj989 testdata.562265 2024-09-18T16:19:19.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:19.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj990 testdata.562265 2024-09-18T16:19:19.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:19.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj991 testdata.562265 2024-09-18T16:19:20.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:20.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj992 testdata.562265 2024-09-18T16:19:20.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:20.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj993 testdata.562265 2024-09-18T16:19:20.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:20.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj994 testdata.562265 2024-09-18T16:19:20.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:20.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj995 testdata.562265 2024-09-18T16:19:20.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:20.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj996 testdata.562265 2024-09-18T16:19:20.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:20.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj997 testdata.562265 2024-09-18T16:19:21.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:21.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj998 testdata.562265 2024-09-18T16:19:21.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:21.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj999 testdata.562265 2024-09-18T16:19:21.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:19:21.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj1000 testdata.562265 2024-09-18T16:19:21.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:317: _scrub_abort: rm -f testdata.562265 2024-09-18T16:19:21.487 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:319: _scrub_abort: get_primary test obj1 2024-09-18T16:19:21.488 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T16:19:21.488 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T16:19:21.488 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T16:19:21.488 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T16:19:21.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:319: _scrub_abort: local primary=1 2024-09-18T16:19:21.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:320: _scrub_abort: local pgid=1.0 2024-09-18T16:19:21.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:322: _scrub_abort: ceph tell 1.0 deep_scrub 2024-09-18T16:19:22.012 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:19:22.013 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": true, 2024-09-18T16:19:22.013 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T16:17:42.010707+0000" 2024-09-18T16:19:22.013 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:19:22.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:324: _scrub_abort: '[' deep_scrub = deep_scrub ']' 2024-09-18T16:19:22.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:326: _scrub_abort: ceph tell 1.0 scrub 2024-09-18T16:19:22.149 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:19:22.149 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T16:19:22.149 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T16:17:42.147565+0000" 2024-09-18T16:19:22.149 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:19:22.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: set -o pipefail 2024-09-18T16:19:22.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:331: _scrub_abort: found=no 2024-09-18T16:19:22.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: seq 0 200 2024-09-18T16:19:22.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: for i in $(seq 0 200) 2024-09-18T16:19:22.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: flush_pg_stats 2024-09-18T16:19:22.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:19:22.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:19:22.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:19:22.460 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:19:22.460 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:19:22.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:19:22.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:19:22.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:19:22.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836519 2024-09-18T16:19:22.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836519 2024-09-18T16:19:22.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836519' 2024-09-18T16:19:22.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:19:22.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:19:22.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672996 2024-09-18T16:19:22.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672996 2024-09-18T16:19:22.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836519 1-42949672996' 2024-09-18T16:19:22.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:19:22.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:19:22.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509474 2024-09-18T16:19:22.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509474 2024-09-18T16:19:22.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836519 1-42949672996 2-64424509474' 2024-09-18T16:19:22.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:19:22.922 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836519 2024-09-18T16:19:22.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:19:22.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:19:22.926 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836519 2024-09-18T16:19:22.926 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:19:22.928 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836519 2024-09-18T16:19:22.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836519 2024-09-18T16:19:22.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2024-09-18T16:19:22.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:19:23.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836519 -lt 21474836519 2024-09-18T16:19:23.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:19:23.230 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672996 2024-09-18T16:19:23.230 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:19:23.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:19:23.233 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672996 2024-09-18T16:19:23.233 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:19:23.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672996 2024-09-18T16:19:23.236 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672996 2024-09-18T16:19:23.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672996' 2024-09-18T16:19:23.237 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:19:23.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672996 -lt 42949672996 2024-09-18T16:19:23.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:19:23.590 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509474 2024-09-18T16:19:23.590 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:19:23.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:19:23.593 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509474 2024-09-18T16:19:23.593 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:19:23.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509474 2024-09-18T16:19:23.596 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509474 2024-09-18T16:19:23.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509474' 2024-09-18T16:19:23.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:19:23.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509473 -lt 64424509474 2024-09-18T16:19:23.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:19:24.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:19:24.902 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:19:25.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509474 -lt 64424509474 2024-09-18T16:19:25.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: ceph pg dump pgs 2024-09-18T16:19:25.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: grep '^1.0' 2024-09-18T16:19:25.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: grep -q scrubbing 2024-09-18T16:19:25.546 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:19:25.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: for i in $(seq 0 200) 2024-09-18T16:19:25.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: flush_pg_stats 2024-09-18T16:19:25.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:19:25.557 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:19:25.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:19:25.852 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:19:25.852 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:19:25.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:19:25.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:19:25.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:19:25.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836521 2024-09-18T16:19:25.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836521 2024-09-18T16:19:25.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521' 2024-09-18T16:19:25.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:19:25.975 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:19:26.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672998 2024-09-18T16:19:26.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672998 2024-09-18T16:19:26.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672998' 2024-09-18T16:19:26.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:19:26.108 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:19:26.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509476 2024-09-18T16:19:26.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509476 2024-09-18T16:19:26.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672998 2-64424509476' 2024-09-18T16:19:26.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:19:26.245 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836521 2024-09-18T16:19:26.245 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:19:26.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:19:26.248 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836521 2024-09-18T16:19:26.248 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:19:26.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836521 2024-09-18T16:19:26.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2024-09-18T16:19:26.250 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836521 2024-09-18T16:19:26.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:19:26.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836519 -lt 21474836521 2024-09-18T16:19:26.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:19:27.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:19:27.557 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:19:27.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836521 -lt 21474836521 2024-09-18T16:19:27.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:19:27.864 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672998 2024-09-18T16:19:27.864 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:19:27.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:19:27.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672998 2024-09-18T16:19:27.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:19:27.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672998 2024-09-18T16:19:27.869 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672998 2024-09-18T16:19:27.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2024-09-18T16:19:27.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:19:28.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672998 -lt 42949672998 2024-09-18T16:19:28.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:19:28.215 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509476 2024-09-18T16:19:28.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:19:28.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:19:28.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509476 2024-09-18T16:19:28.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:19:28.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509476 2024-09-18T16:19:28.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509476' 2024-09-18T16:19:28.221 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509476 2024-09-18T16:19:28.222 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:19:28.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509476 -lt 64424509476 2024-09-18T16:19:28.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: ceph pg dump pgs 2024-09-18T16:19:28.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: grep '^1.0' 2024-09-18T16:19:28.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: grep -q scrubbing 2024-09-18T16:19:28.902 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:19:28.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: found=yes 2024-09-18T16:19:28.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: break 2024-09-18T16:19:28.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:342: _scrub_abort: set +o pipefail 2024-09-18T16:19:28.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:344: _scrub_abort: test yes = no 2024-09-18T16:19:28.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:350: _scrub_abort: ceph osd set nodeep-scrub 2024-09-18T16:19:29.349 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is set 2024-09-18T16:19:29.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: '[' deep_scrub = deep_scrub ']' 2024-09-18T16:19:29.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:353: _scrub_abort: ceph osd set noscrub 2024-09-18T16:19:29.919 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T16:19:29.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:357: _scrub_abort: set -o pipefail 2024-09-18T16:19:29.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:358: _scrub_abort: seq 0 200 2024-09-18T16:19:29.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:358: _scrub_abort: for i in $(seq 0 200) 2024-09-18T16:19:29.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:360: _scrub_abort: flush_pg_stats 2024-09-18T16:19:29.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:19:29.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:19:30.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:19:30.238 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:19:30.238 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:19:30.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:19:30.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:19:30.239 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:19:30.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836522 2024-09-18T16:19:30.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836522 2024-09-18T16:19:30.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836522' 2024-09-18T16:19:30.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:19:30.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:19:30.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673000 2024-09-18T16:19:30.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673000 2024-09-18T16:19:30.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836522 1-42949673000' 2024-09-18T16:19:30.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:19:30.506 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:19:30.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509478 2024-09-18T16:19:30.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509478 2024-09-18T16:19:30.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836522 1-42949673000 2-64424509478' 2024-09-18T16:19:30.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:19:30.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836522 2024-09-18T16:19:30.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:19:30.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:19:30.651 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836522 2024-09-18T16:19:30.651 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:19:30.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836522 2024-09-18T16:19:30.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836522' 2024-09-18T16:19:30.652 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836522 2024-09-18T16:19:30.653 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:19:30.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836523 -lt 21474836522 2024-09-18T16:19:30.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:19:30.962 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673000 2024-09-18T16:19:30.962 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:19:30.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:19:30.966 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673000 2024-09-18T16:19:30.967 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:19:30.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673000 2024-09-18T16:19:30.969 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949673000 2024-09-18T16:19:30.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673000' 2024-09-18T16:19:30.970 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:19:31.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673000 -lt 42949673000 2024-09-18T16:19:31.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:19:31.321 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509478 2024-09-18T16:19:31.321 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:19:31.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:19:31.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509478 2024-09-18T16:19:31.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:19:31.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509478 2024-09-18T16:19:31.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509478' 2024-09-18T16:19:31.326 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509478 2024-09-18T16:19:31.327 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:19:31.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509478 -lt 64424509478 2024-09-18T16:19:31.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:361: _scrub_abort: ceph pg dump pgs 2024-09-18T16:19:31.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:361: _scrub_abort: grep '^1.0' 2024-09-18T16:19:31.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:361: _scrub_abort: grep -q scrubbing 2024-09-18T16:19:31.956 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:19:31.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:363: _scrub_abort: continue 2024-09-18T16:19:31.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:358: _scrub_abort: for i in $(seq 0 200) 2024-09-18T16:19:31.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:360: _scrub_abort: flush_pg_stats 2024-09-18T16:19:31.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:19:31.966 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:19:32.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:19:32.259 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:19:32.260 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:19:32.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:19:32.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:19:32.260 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:19:32.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836524 2024-09-18T16:19:32.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836524 2024-09-18T16:19:32.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524' 2024-09-18T16:19:32.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:19:32.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:19:32.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673001 2024-09-18T16:19:32.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673001 2024-09-18T16:19:32.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949673001' 2024-09-18T16:19:32.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:19:32.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:19:32.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509479 2024-09-18T16:19:32.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509479 2024-09-18T16:19:32.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949673001 2-64424509479' 2024-09-18T16:19:32.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:19:32.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836524 2024-09-18T16:19:32.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:19:32.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:19:32.725 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836524 2024-09-18T16:19:32.725 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:19:32.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836524 2024-09-18T16:19:32.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-09-18T16:19:32.728 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836524 2024-09-18T16:19:32.728 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:19:33.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836524 -lt 21474836524 2024-09-18T16:19:33.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:19:33.033 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673001 2024-09-18T16:19:33.034 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:19:33.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:19:33.037 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673001 2024-09-18T16:19:33.038 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:19:33.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673001 2024-09-18T16:19:33.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673001' 2024-09-18T16:19:33.040 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949673001 2024-09-18T16:19:33.041 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:19:33.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673001 -lt 42949673001 2024-09-18T16:19:33.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:19:33.374 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509479 2024-09-18T16:19:33.374 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:19:33.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:19:33.377 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509479 2024-09-18T16:19:33.377 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:19:33.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509479 2024-09-18T16:19:33.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509479' 2024-09-18T16:19:33.379 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509479 2024-09-18T16:19:33.380 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:19:33.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509479 -lt 64424509479 2024-09-18T16:19:33.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:361: _scrub_abort: ceph pg dump pgs 2024-09-18T16:19:33.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:361: _scrub_abort: grep '^1.0' 2024-09-18T16:19:33.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:361: _scrub_abort: grep -q scrubbing 2024-09-18T16:19:34.011 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:19:34.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:363: _scrub_abort: continue 2024-09-18T16:19:34.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:358: _scrub_abort: for i in $(seq 0 200) 2024-09-18T16:19:34.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:360: _scrub_abort: flush_pg_stats 2024-09-18T16:19:34.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:19:34.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:19:34.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:19:34.328 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:19:34.328 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:19:34.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:19:34.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:19:34.328 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:19:34.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836525 2024-09-18T16:19:34.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836525 2024-09-18T16:19:34.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836525' 2024-09-18T16:19:34.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:19:34.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:19:34.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673003 2024-09-18T16:19:34.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673003 2024-09-18T16:19:34.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836525 1-42949673003' 2024-09-18T16:19:34.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:19:34.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:19:34.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509480 2024-09-18T16:19:34.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509480 2024-09-18T16:19:34.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836525 1-42949673003 2-64424509480' 2024-09-18T16:19:34.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:19:34.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836525 2024-09-18T16:19:34.814 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:19:34.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:19:34.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836525 2024-09-18T16:19:34.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:19:34.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836525 2024-09-18T16:19:34.819 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836525 2024-09-18T16:19:34.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2024-09-18T16:19:34.820 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:19:35.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836525 -lt 21474836525 2024-09-18T16:19:35.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:19:35.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673003 2024-09-18T16:19:35.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:19:35.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:19:35.127 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673003 2024-09-18T16:19:35.127 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:19:35.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673003 2024-09-18T16:19:35.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673003' 2024-09-18T16:19:35.129 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949673003 2024-09-18T16:19:35.130 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:19:35.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673003 -lt 42949673003 2024-09-18T16:19:35.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:19:35.476 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509480 2024-09-18T16:19:35.476 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:19:35.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:19:35.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509480 2024-09-18T16:19:35.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:19:35.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509480 2024-09-18T16:19:35.481 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509480 2024-09-18T16:19:35.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509480' 2024-09-18T16:19:35.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:19:35.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509480 -lt 64424509480 2024-09-18T16:19:35.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:361: _scrub_abort: ceph pg dump pgs 2024-09-18T16:19:35.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:361: _scrub_abort: grep '^1.0' 2024-09-18T16:19:35.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:361: _scrub_abort: grep -q scrubbing 2024-09-18T16:19:36.197 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:19:36.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:366: _scrub_abort: break 2024-09-18T16:19:36.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:368: _scrub_abort: set +o pipefail 2024-09-18T16:19:36.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: sleep 5 2024-09-18T16:19:41.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: grep 'nodeep_scrub set, aborting' td/osd-scrub-test/osd.1.log 2024-09-18T16:19:41.228 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T16:19:33.879+0000 7f305623e700 10 osd.1 pg_epoch: 21 pg[1.0( v 19'1000 (0'0,19'1000] local-lis/les=17/18 n=1000 ec=17/17 lis/c=17/17 les/c/f=18/18/0 sis=17) [1,0,2] r=0 lpr=17 crt=19'1000 lcod 19'999 mlcod 19'999 active+clean+scrubbing+deep [ 1.0: ] TIME_FOR_DEEP] scrubber : nodeep_scrub set, aborting 2024-09-18T16:19:41.230 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: get_last_scrub_stamp 1.0 2024-09-18T16:19:41.231 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:19:41.231 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:19:41.231 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:19:41.231 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:19:41.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: local last_scrub=2024-09-11T16:17:42.147565+0000 2024-09-18T16:19:41.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: ceph config set osd osd_scrub_sleep 0.1 2024-09-18T16:19:42.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:381: _scrub_abort: ceph osd unset nodeep-scrub 2024-09-18T16:19:42.522 INFO:tasks.workunit.client.0.smithi046.stderr:nodeep-scrub is unset 2024-09-18T16:19:42.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: '[' deep_scrub = deep_scrub ']' 2024-09-18T16:19:42.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:384: _scrub_abort: ceph osd unset noscrub 2024-09-18T16:19:43.042 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is unset 2024-09-18T16:19:43.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:386: _scrub_abort: TIMEOUT=500 2024-09-18T16:19:43.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:387: _scrub_abort: wait_for_scrub 1.0 2024-09-11T16:17:42.147565+0000 2024-09-18T16:19:43.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-18T16:19:43.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-11T16:17:42.147565+0000 2024-09-18T16:19:43.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T16:19:43.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T16:19:43.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:19:43.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:19:43.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:19:43.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:19:43.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:19:43.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:19:43.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:19:43.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:19:44.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:19:44.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:19:44.395 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:19:44.395 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:19:44.395 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:19:44.395 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:19:44.395 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:19:44.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:19:44.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:19:45.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:19:45.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:19:45.704 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:19:45.704 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:19:45.704 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:19:45.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:19:45.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:19:46.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:19:46.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:19:47.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:19:47.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:19:47.013 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:19:47.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:19:47.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:19:47.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:19:47.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:19:47.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:19:47.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:19:48.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:19:48.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:19:48.321 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:19:48.321 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:19:48.321 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:19:48.321 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:19:48.322 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:19:48.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:19:48.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:19:49.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:19:49.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:19:49.626 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:19:49.626 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:19:49.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:19:49.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:19:49.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:19:49.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:19:49.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:19:50.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:19:50.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:19:50.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:19:50.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:19:50.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:19:50.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:19:50.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:19:51.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:19:51.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:19:52.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:19:52.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:19:52.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:19:52.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:19:52.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:19:52.242 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:19:52.242 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:19:52.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:19:52.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:19:53.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:19:53.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:19:53.553 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:19:53.553 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:19:53.553 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:19:53.553 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:19:53.554 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:19:53.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:19:53.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:19:54.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:19:54.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:19:54.879 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:19:54.879 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:19:54.879 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:19:54.879 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:19:54.880 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:19:55.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:19:55.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:19:56.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:19:56.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:19:56.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:19:56.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:19:56.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:19:56.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:19:56.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:19:56.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:19:56.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:19:57.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:19:57.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:19:57.492 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:19:57.492 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:19:57.492 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:19:57.492 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:19:57.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:19:57.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:19:57.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:19:58.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:19:58.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:19:58.849 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:19:58.849 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:19:58.849 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:19:58.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:19:58.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:19:59.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:19:59.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:00.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:00.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:00.159 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:00.159 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:00.159 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:00.159 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:00.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:00.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:00.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:01.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:01.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:01.479 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:01.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:01.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:01.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:01.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:01.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:01.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:02.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:02.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:02.796 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:02.797 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:02.797 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:02.797 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:02.797 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:03.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:03.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:04.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:04.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:04.101 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:04.101 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:04.101 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:04.102 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:04.102 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:04.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:04.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:05.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:05.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:05.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:05.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:05.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:05.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:05.414 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:05.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:05.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:06.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:06.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:06.712 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:06.712 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:06.712 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:06.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:06.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:07.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:07.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:08.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:08.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:08.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:08.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:08.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:08.032 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:08.032 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:08.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:08.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:09.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:09.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:09.336 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:09.336 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:09.336 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:09.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:09.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:09.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:09.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:10.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:10.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:10.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:10.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:10.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:10.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:10.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:10.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:10.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:11.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:11.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:11.948 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:11.948 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:11.948 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:11.949 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:11.949 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:12.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:12.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:13.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:13.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:13.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:13.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:13.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:13.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:13.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:13.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:13.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:14.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:14.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:14.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:14.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:14.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:14.576 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:14.576 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:14.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:14.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:15.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:15.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:15.881 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:15.881 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:15.881 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:15.882 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:15.882 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:16.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:16.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:17.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:17.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:17.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:17.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:17.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:17.192 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:17.192 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:17.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:17.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:18.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:18.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:18.498 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:18.498 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:18.498 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:18.499 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:18.499 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:18.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:18.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:19.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:19.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:19.811 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:19.811 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:19.811 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:19.811 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:19.811 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:20.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:20.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:21.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:21.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:21.120 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:21.120 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:21.120 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:21.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:21.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:21.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:21.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:22.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:22.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:22.428 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:22.428 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:22.429 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:22.429 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:22.429 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:22.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:22.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:23.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:23.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:23.742 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:23.742 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:23.742 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:23.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:23.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:24.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:24.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:25.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:25.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:25.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:25.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:25.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:25.055 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:25.055 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:25.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:25.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:26.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:26.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:26.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:26.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:26.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:26.369 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:26.369 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:26.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:26.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:27.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:27.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:27.684 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:27.684 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:27.684 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:27.685 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:27.685 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:27.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:28.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:29.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:29.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:29.003 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:29.003 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:29.003 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:29.004 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:29.004 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:29.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:29.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:30.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:30.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:30.311 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:30.311 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:30.311 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:30.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:30.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:30.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:30.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:31.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:31.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:31.622 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:31.622 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:31.622 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:31.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:31.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:31.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:31.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:32.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:32.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:32.925 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:32.925 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:32.925 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:32.925 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:32.926 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:33.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:33.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:34.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:34.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:34.234 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:34.234 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:34.234 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:34.235 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:34.235 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:34.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:34.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:35.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:35.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:35.549 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:35.549 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:35.549 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:35.550 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:35.550 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:35.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:35.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:36.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:36.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:36.855 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:36.855 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:36.855 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:36.856 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:36.856 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:37.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:37.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:38.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:38.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:38.165 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:38.165 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:38.165 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:38.165 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:38.166 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:38.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:38.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:39.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:39.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:39.472 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:39.473 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:39.473 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:39.473 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:39.473 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:39.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:39.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:40.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:40.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:40.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:40.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:40.796 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:40.796 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:40.796 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:41.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:41.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:42.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:42.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:42.104 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:42.104 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:42.104 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:42.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:42.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:42.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:42.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:43.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:43.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:43.418 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:43.418 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:43.418 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:43.418 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:43.418 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:43.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:43.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:44.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:44.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:44.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:44.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:44.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:44.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:44.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:45.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:45.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:46.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:46.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:46.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:46.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:46.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:46.032 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:46.032 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:46.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:46.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:47.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:47.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:47.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:47.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:47.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:47.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:47.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:47.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:47.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:48.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:48.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:48.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:48.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:48.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:48.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:48.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:48.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:48.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:49.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:49.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:49.965 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:49.965 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:49.965 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:49.966 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:49.966 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:50.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:50.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:51.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:51.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:51.270 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:51.270 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:51.270 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:51.271 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:51.271 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:51.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:51.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:52.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:52.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:52.580 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:52.580 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:52.580 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:52.580 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:52.580 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:52.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:52.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:53.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:53.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:53.897 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:53.897 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:53.897 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:53.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:53.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:54.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:54.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:55.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:55.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:55.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:55.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:55.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:55.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:55.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:55.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:55.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:56.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:56.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:56.524 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:56.524 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:56.524 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:56.524 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:56.525 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:56.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:56.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:57.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:57.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:57.827 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:57.828 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:57.828 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:57.828 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:57.828 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:58.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:58.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:20:59.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:20:59.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:20:59.133 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:20:59.133 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:20:59.133 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:20:59.134 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:20:59.134 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:20:59.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:20:59.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:00.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:00.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:00.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:00.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:00.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:00.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:00.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:00.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:00.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:01.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:01.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:01.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:01.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:01.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:01.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:01.744 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:02.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:02.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:03.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:03.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:03.055 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:03.055 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:03.055 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:03.055 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:03.056 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:03.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:03.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:04.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:04.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:04.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:04.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:04.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:04.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:04.364 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:04.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:04.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:05.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:05.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:05.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:05.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:05.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:05.675 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:05.675 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:05.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:05.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:06.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:06.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:06.988 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:06.989 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:06.989 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:06.989 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:06.989 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:07.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:07.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:08.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:08.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:08.301 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:08.301 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:08.301 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:08.301 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:08.302 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:08.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:08.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:09.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:09.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:09.612 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:09.612 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:09.612 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:09.612 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:09.612 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:09.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:09.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:10.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:10.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:10.922 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:10.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:10.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:10.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:10.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:11.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:11.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:12.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:12.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:12.239 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:12.239 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:12.239 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:12.240 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:12.240 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:12.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:12.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:13.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:13.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:13.548 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:13.548 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:13.548 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:13.548 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:13.549 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:13.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:13.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:14.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:14.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:14.868 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:14.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:14.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:14.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:14.871 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:15.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:15.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:16.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:16.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:16.180 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:16.180 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:16.180 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:16.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:16.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:16.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:16.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:17.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:17.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:17.490 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:17.490 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:17.490 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:17.490 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:17.490 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:17.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:17.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:18.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:18.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:18.798 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:18.798 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:18.798 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:18.798 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:18.799 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:19.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:19.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:20.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:20.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:20.116 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:20.116 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:20.116 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:20.116 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:20.116 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:20.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:20.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:21.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:21.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:21.421 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:21.421 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:21.421 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:21.422 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:21.422 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:21.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:21.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:22.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:22.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:22.734 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:22.734 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:22.734 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:22.734 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:22.734 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:23.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:23.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:24.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:24.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:24.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:24.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:24.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:24.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:24.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:24.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:24.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:25.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:25.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:25.359 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:25.359 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:25.359 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:25.359 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:25.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:25.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:25.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:26.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:26.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:26.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:26.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:26.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:26.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:26.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:26.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:26.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:27.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:27.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:27.976 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:27.976 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:27.976 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:27.977 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:27.977 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:28.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:28.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:29.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:29.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:29.286 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:29.286 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:29.286 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:29.287 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:29.287 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:29.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:29.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:30.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:30.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:30.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:30.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:30.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:30.592 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:30.592 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:30.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:30.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:31.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:31.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:31.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:31.902 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:31.902 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:31.902 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:31.902 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:32.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:32.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:33.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:33.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:33.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:33.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:33.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:33.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:33.214 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:33.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:33.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:34.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:34.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:34.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:34.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:34.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:34.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:34.520 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:34.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:34.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:35.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:35.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:35.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:35.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:35.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:35.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:35.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:36.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:36.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:37.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:37.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:37.142 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:37.143 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:37.143 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:37.143 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:37.143 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:37.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:37.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:38.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:38.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:38.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:38.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:38.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:38.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:38.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:38.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:38.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:39.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:39.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:39.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:39.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:39.776 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:39.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:39.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:40.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:40.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:41.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:41.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:41.092 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:41.092 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:41.092 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:41.093 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:41.093 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:41.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:41.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:42.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:42.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:42.400 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:42.400 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:42.400 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:42.401 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:42.401 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:42.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:42.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:43.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:43.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:43.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:43.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:43.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:43.712 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:43.712 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:44.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:44.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:45.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:45.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:45.021 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:45.021 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:45.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:45.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:45.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:45.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:45.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:46.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:46.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:46.336 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:46.336 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:46.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:46.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:46.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:46.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:46.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:47.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:47.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:47.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:47.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:47.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:47.648 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:47.648 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:47.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:47.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:48.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:48.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:48.954 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:48.954 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:48.954 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:48.954 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:48.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:49.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:49.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:50.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:50.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:50.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:50.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:50.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:50.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:50.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:50.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:50.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:51.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:51.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:51.566 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:51.566 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:51.566 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:51.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:51.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:51.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:51.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:52.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:52.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:52.887 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:52.887 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:52.887 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:52.887 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:52.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:53.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:53.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:54.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:54.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:54.197 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:54.197 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:54.197 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:54.197 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:54.198 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:54.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:54.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:55.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:55.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:55.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:55.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:55.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:55.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:55.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:55.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:55.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:56.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:56.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:56.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:56.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:56.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:56.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:56.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:57.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:57.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:58.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:58.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:58.133 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:58.133 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:58.133 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:58.133 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:58.134 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:58.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:58.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:21:59.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:21:59.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:21:59.437 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:21:59.437 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:21:59.437 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:21:59.437 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:21:59.437 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:21:59.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:21:59.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:00.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:00.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:00.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:00.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:00.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:00.753 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:00.753 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:01.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:01.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:02.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:02.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:02.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:02.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:02.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:02.067 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:02.067 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:02.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:02.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:03.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:03.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:03.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:03.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:03.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:03.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:03.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:03.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:03.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:04.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:04.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:04.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:04.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:04.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:04.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:04.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:05.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:05.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:06.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:06.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:06.013 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:06.013 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:06.013 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:06.013 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:06.013 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:06.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:06.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:07.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:07.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:07.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:07.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:07.320 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:07.320 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:07.320 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:07.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:07.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:08.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:08.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:08.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:08.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:08.642 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:08.642 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:08.642 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:08.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:08.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:09.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:09.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:09.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:09.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:09.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:09.956 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:09.956 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:10.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:10.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:11.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:11.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:11.267 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:11.267 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:11.267 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:11.268 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:11.268 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:11.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:11.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:12.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:12.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:12.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:12.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:12.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:12.583 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:12.583 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:12.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:12.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:13.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:13.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:13.902 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:13.902 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:13.902 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:13.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:13.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:14.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:14.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:15.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:15.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:15.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:15.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:15.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:15.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:15.209 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:15.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:15.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:16.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:16.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:16.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:16.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:16.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:16.520 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:16.520 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:16.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:16.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:17.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:17.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:17.827 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:17.827 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:17.827 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:17.828 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:17.828 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:18.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:18.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:19.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:19.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:19.145 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:19.145 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:19.146 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:19.146 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:19.146 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:19.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:19.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:20.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:20.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:20.452 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:20.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:20.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:20.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:20.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:20.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:20.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:21.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:21.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:21.768 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:21.768 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:21.768 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:21.769 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:21.769 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:22.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:22.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:23.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:23.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:23.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:23.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:23.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:23.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:23.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:23.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:23.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:24.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:24.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:24.393 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:24.393 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:24.393 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:24.394 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:24.394 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:24.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:24.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:25.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:25.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:25.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:25.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:25.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:25.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:25.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:26.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:26.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:27.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:27.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:27.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:27.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:27.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:27.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:27.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:27.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:27.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:28.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:28.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:28.322 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:28.322 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:28.322 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:28.322 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:28.323 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:28.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:28.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:29.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:29.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:29.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:29.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:29.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:29.629 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:29.629 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:29.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:29.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:30.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:30.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:30.943 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:30.943 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:30.943 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:30.944 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:30.944 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:31.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:31.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:32.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:32.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:32.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:32.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:32.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:32.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:32.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:32.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:32.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:33.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:33.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:33.564 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:33.564 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:33.564 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:33.565 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:33.565 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:33.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:33.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:34.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:34.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:34.891 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:34.891 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:34.891 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:34.891 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:34.891 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:35.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:35.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:36.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:36.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:36.196 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:36.196 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:36.196 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:36.196 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:36.197 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:36.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:36.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:37.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:37.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:37.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:37.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:37.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:37.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:37.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:37.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:37.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:38.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:38.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:38.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:38.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:38.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:38.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:38.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:39.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:39.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:40.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:40.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:40.138 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:40.138 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:40.138 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:40.138 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:40.138 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:40.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:40.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:41.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:41.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:41.452 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:41.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:41.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:41.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:41.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:41.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:41.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:42.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:42.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:42.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:42.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:42.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:42.762 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:42.762 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:43.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:43.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:44.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:44.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:44.076 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:44.076 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:44.077 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:44.077 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:44.077 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:44.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:44.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:45.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:45.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:45.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:45.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:45.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:45.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:45.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:45.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:45.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:46.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:46.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:46.696 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:46.696 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:46.696 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:46.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:46.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:47.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:47.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:48.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:48.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:48.004 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:48.004 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:48.004 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:48.004 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:48.004 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:48.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:48.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:49.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:49.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:49.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:49.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:49.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:49.315 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:49.315 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:49.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:49.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:50.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:50.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:50.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:50.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:50.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:50.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:50.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:50.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:50.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:51.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:51.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:51.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:51.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:51.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:51.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:51.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:52.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:52.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:53.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:53.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:53.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:53.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:53.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:53.242 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:53.242 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:53.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:53.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:54.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:54.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:54.547 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:54.547 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:54.547 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:54.547 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:54.548 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:54.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:54.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:55.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:55.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:55.856 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:55.856 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:55.856 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:55.856 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:55.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:56.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:56.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:57.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:57.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:57.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:57.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:57.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:57.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:57.165 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:57.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:57.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:58.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:58.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:58.476 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:58.476 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:58.476 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:58.477 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:58.477 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:22:58.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:22:58.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:22:59.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:22:59.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:22:59.784 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:22:59.784 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:22:59.784 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:22:59.785 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:22:59.785 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:23:00.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:23:00.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:23:01.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:23:01.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:23:01.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:23:01.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:23:01.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:23:01.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:23:01.098 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:23:01.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:23:01.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:23:02.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:23:02.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:23:02.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:23:02.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:23:02.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:23:02.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:23:02.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:23:02.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:23:02.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:23:03.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:23:03.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:23:03.715 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:23:03.715 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:23:03.715 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:23:03.716 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:23:03.716 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:23:04.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:23:04.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:23:05.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:23:05.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:23:05.024 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:23:05.024 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:23:05.025 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:23:05.025 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:23:05.025 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:23:05.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:23:05.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:23:06.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:23:06.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:23:06.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:23:06.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:23:06.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:23:06.338 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:23:06.338 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:23:06.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:23:06.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:23:07.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:23:07.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:23:07.645 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:23:07.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:23:07.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:23:07.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:23:07.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:23:07.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:23:07.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:23:08.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:23:08.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:23:08.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:23:08.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:23:08.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:23:08.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:23:08.956 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:23:09.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:23:09.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:23:10.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:23:10.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:23:10.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:23:10.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:23:10.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:23:10.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:23:10.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:23:10.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:23:10.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:23:11.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:23:11.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:23:11.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:23:11.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:23:11.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:23:11.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:23:11.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:23:11.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:23:11.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:23:12.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:23:12.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:23:12.883 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:23:12.883 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:23:12.883 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:23:12.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:23:12.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:23:13.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:23:13.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:23:14.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:23:14.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:23:14.202 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:23:14.202 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:23:14.202 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:23:14.203 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:23:14.203 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:23:14.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:17:42.147565+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:23:14.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:23:15.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:23:15.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:23:15.513 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:23:15.513 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:23:15.513 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:23:15.513 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:23:15.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:23:15.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:23:11.463903+0000 '>' 2024-09-11T16:17:42.147565+0000 2024-09-18T16:23:15.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T16:23:15.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: teardown td/osd-scrub-test 2024-09-18T16:23:15.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-09-18T16:23:15.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:23:15.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-18T16:23:15.836 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:23:15.836 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:23:15.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:23:15.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:23:15.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:23:15.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:23:15.956 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:23:15.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:23:15.960 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:23:15.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:23:15.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:23:15.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:23:15.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:23:15.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:23:15.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:23:15.963 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:23:15.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:23:15.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:23:15.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-09-18T16:23:16.013 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:23:16.013 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:23:16.013 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:23:16.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.562265 2024-09-18T16:23:16.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:23:16.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:23:16.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-09-18T16:23:16.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: setup td/osd-scrub-test 2024-09-18T16:23:16.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-test 2024-09-18T16:23:16.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-test 2024-09-18T16:23:16.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-09-18T16:23:16.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:23:16.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-18T16:23:16.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:23:16.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:23:16.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:23:16.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:23:16.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:23:16.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:23:16.021 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:23:16.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:23:16.024 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:23:16.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:23:16.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:23:16.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:23:16.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:23:16.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:23:16.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:23:16.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:23:16.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:23:16.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:23:16.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-09-18T16:23:16.035 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:23:16.035 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:23:16.035 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:23:16.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.562265 2024-09-18T16:23:16.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:23:16.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:23:16.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-test 2024-09-18T16:23:16.039 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T16:23:16.039 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:23:16.039 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:23:16.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.562265 2024-09-18T16:23:16.042 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T16:23:16.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T16:23:16.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T16:23:16.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-09-18T16:23:16.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: TEST_dump_scrub_schedule td/osd-scrub-test 2024-09-18T16:23:16.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:505: TEST_dump_scrub_schedule: local dir=td/osd-scrub-test 2024-09-18T16:23:16.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:506: TEST_dump_scrub_schedule: local poolname=test 2024-09-18T16:23:16.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:507: TEST_dump_scrub_schedule: local OSDS=3 2024-09-18T16:23:16.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:508: TEST_dump_scrub_schedule: local objects=15 2024-09-18T16:23:16.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:510: TEST_dump_scrub_schedule: TESTDATA=testdata.562265 2024-09-18T16:23:16.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:512: TEST_dump_scrub_schedule: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2024-09-18T16:23:16.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-test 2024-09-18T16:23:16.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T16:23:16.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T16:23:16.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T16:23:16.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-test/a 2024-09-18T16:23:16.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2024-09-18T16:23:16.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T16:23:16.426 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:23:16.426 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:23:16.426 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:23:16.426 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:23:16.426 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:23:16.427 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:23:16.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-18T16:23:16.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T16:23:16.716 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T16:23:16.716 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:23:16.716 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:23:16.716 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T16:23:16.716 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T16:23:16.717 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:23:16.717 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:23:16.717 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:23:16.718 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:23:16.718 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:23:16.718 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:23:16.719 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.562265/ceph-mon.a.asok 2024-09-18T16:23:16.720 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:23:16.720 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.562265/ceph-mon.a.asok config get fsid 2024-09-18T16:23:16.810 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T16:23:16.811 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:23:16.811 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:23:16.811 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T16:23:16.811 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T16:23:16.812 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:23:16.812 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:23:16.812 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:23:16.812 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:23:16.812 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:23:16.813 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:23:16.813 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.562265/ceph-mon.a.asok 2024-09-18T16:23:16.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:23:16.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.562265/ceph-mon.a.asok config get mon_host 2024-09-18T16:23:16.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:513: TEST_dump_scrub_schedule: run_mgr td/osd-scrub-test x 2024-09-18T16:23:16.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-test 2024-09-18T16:23:16.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T16:23:16.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T16:23:16.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T16:23:16.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-test/x 2024-09-18T16:23:16.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T16:23:17.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T16:23:17.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:23:17.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:23:17.773 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:23:17.773 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:23:17.773 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:23:17.774 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:23:17.774 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:23:17.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:23:17.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:522: TEST_dump_scrub_schedule: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_scrub_sleep=0.2' 2024-09-18T16:23:17.808 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:524: TEST_dump_scrub_schedule: expr 3 - 1 2024-09-18T16:23:17.810 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:524: TEST_dump_scrub_schedule: seq 0 2 2024-09-18T16:23:17.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:524: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:23:17.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:526: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 0 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_scrub_sleep=0.2 2024-09-18T16:23:17.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-09-18T16:23:17.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:23:17.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T16:23:17.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:23:17.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/0 2024-09-18T16:23:17.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-18T16:23:17.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:23:17.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:23:17.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:23:17.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-09-18T16:23:17.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-09-18T16:23:17.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:23:17.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:23:17.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-18T16:23:17.815 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:23:17.815 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:23:17.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:23:17.816 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:23:17.816 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:23:17.816 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:23:17.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:23:17.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:23:17.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:23:17.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:23:17.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:23:17.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-18T16:23:17.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-18T16:23:17.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:23:17.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:23:17.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:23:17.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:23:17.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:23:17.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_scrub_sleep=0.2' 2024-09-18T16:23:17.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/0 2024-09-18T16:23:17.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:23:17.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=50942d41-b1ec-41c8-aa7b-91052cea2132 2024-09-18T16:23:17.818 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 50942d41-b1ec-41c8-aa7b-91052cea2132 2024-09-18T16:23:17.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 50942d41-b1ec-41c8-aa7b-91052cea2132' 2024-09-18T16:23:17.818 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:23:17.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB1/upmdRCDMRAAF+rUkcVpgEyOXQi95ifgNA== 2024-09-18T16:23:17.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB1/upmdRCDMRAAF+rUkcVpgEyOXQi95ifgNA=="}' 2024-09-18T16:23:17.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 50942d41-b1ec-41c8-aa7b-91052cea2132 -i td/osd-scrub-test/0/new.json 2024-09-18T16:23:18.116 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:23:18.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/0/new.json 2024-09-18T16:23:18.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_scrub_sleep=0.2 --mkfs --key AQB1/upmdRCDMRAAF+rUkcVpgEyOXQi95ifgNA== --osd-uuid 50942d41-b1ec-41c8-aa7b-91052cea2132 2024-09-18T16:23:18.156 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:18.148+0000 7fb78fbef540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:23:18.159 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:18.152+0000 7fb78fbef540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:23:18.161 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:18.152+0000 7fb78fbef540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:23:18.162 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:18.152+0000 7fb78fbef540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:23:18.162 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:18.152+0000 7fb78fbef540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:23:18.162 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:18.152+0000 7fb78fbef540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:23:18.240 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:18.232+0000 7fb78fbef540 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-09-18T16:23:24.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-09-18T16:23:24.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:23:24.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T16:23:24.366 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T16:23:24.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:23:24.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T16:23:24.761 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T16:23:24.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_scrub_sleep=0.2 2024-09-18T16:23:24.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:23:24.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:23:24.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:23:24.793 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:24.788+0000 7f70ab871540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:23:24.796 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:24.788+0000 7f70ab871540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:23:24.798 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:24.792+0000 7f70ab871540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:23:25.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T16:23:25.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:23:25.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:23:25.054 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:23:25.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:23:25.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:23:25.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:23:25.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:23:25.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:23:25.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:23:25.348 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:25.340+0000 7f70ab871540 -1 Falling back to public interface 2024-09-18T16:23:25.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:23:26.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:23:26.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:23:26.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:23:26.376 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:23:26.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:23:26.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:23:26.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:23:27.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:23:27.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:23:27.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:23:27.688 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:23:27.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:23:27.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:23:28.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:23:28.037 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:28.028+0000 7f70ab871540 -1 osd.0 0 log_to_monitors true 2024-09-18T16:23:29.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:23:29.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:23:29.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:23:29.003 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:23:29.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:23:29.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:23:29.369 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/593851,v1:127.0.0.1:6803/593851] [v2:127.0.0.1:6804/593851,v1:127.0.0.1:6805/593851] exists,up 50942d41-b1ec-41c8-aa7b-91052cea2132 2024-09-18T16:23:29.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:23:29.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:23:29.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:23:29.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:524: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:23:29.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:526: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 1 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_scrub_sleep=0.2 2024-09-18T16:23:29.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-09-18T16:23:29.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:23:29.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T16:23:29.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:23:29.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/1 2024-09-18T16:23:29.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-18T16:23:29.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:23:29.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:23:29.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:23:29.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-09-18T16:23:29.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-09-18T16:23:29.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:23:29.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:23:29.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-18T16:23:29.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:23:29.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:23:29.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:23:29.375 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:23:29.375 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:23:29.375 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:23:29.375 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:23:29.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:23:29.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:23:29.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:23:29.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:23:29.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-18T16:23:29.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-18T16:23:29.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:23:29.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:23:29.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:23:29.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:23:29.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:23:29.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_scrub_sleep=0.2' 2024-09-18T16:23:29.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/1 2024-09-18T16:23:29.376 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:23:29.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e70875d3-dab0-4a0a-9227-dfaf81bd860c 2024-09-18T16:23:29.378 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 e70875d3-dab0-4a0a-9227-dfaf81bd860c 2024-09-18T16:23:29.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e70875d3-dab0-4a0a-9227-dfaf81bd860c' 2024-09-18T16:23:29.378 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:23:29.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCB/upms4ZmFxAAjPQfRUuIncqnzxmNCF9AhQ== 2024-09-18T16:23:29.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCB/upms4ZmFxAAjPQfRUuIncqnzxmNCF9AhQ=="}' 2024-09-18T16:23:29.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e70875d3-dab0-4a0a-9227-dfaf81bd860c -i td/osd-scrub-test/1/new.json 2024-09-18T16:23:29.827 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:23:29.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/1/new.json 2024-09-18T16:23:29.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_scrub_sleep=0.2 --mkfs --key AQCB/upms4ZmFxAAjPQfRUuIncqnzxmNCF9AhQ== --osd-uuid e70875d3-dab0-4a0a-9227-dfaf81bd860c 2024-09-18T16:23:29.883 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:29.876+0000 7f19c688a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:23:29.886 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:29.880+0000 7f19c688a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:23:29.888 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:29.880+0000 7f19c688a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:23:29.889 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:29.880+0000 7f19c688a540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:23:29.889 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:29.880+0000 7f19c688a540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:23:29.889 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:29.880+0000 7f19c688a540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:23:29.948 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:29.940+0000 7f19c688a540 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-09-18T16:23:35.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-09-18T16:23:35.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:23:35.895 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T16:23:35.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T16:23:35.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:23:36.279 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T16:23:36.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T16:23:36.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_scrub_sleep=0.2 2024-09-18T16:23:36.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:23:36.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:23:36.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:23:36.310 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:36.304+0000 7fab8c540540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:23:36.313 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:36.304+0000 7fab8c540540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:23:36.314 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:36.308+0000 7fab8c540540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:23:36.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T16:23:36.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:23:36.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T16:23:36.622 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:23:36.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:23:36.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:23:36.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:23:36.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:23:36.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:23:36.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:23:36.847 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:36.840+0000 7fab8c540540 -1 Falling back to public interface 2024-09-18T16:23:36.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:23:37.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:23:37.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:23:37.975 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:23:37.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:23:37.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:23:37.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:23:38.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:23:39.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:23:39.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:23:39.276 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:23:39.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:23:39.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:23:39.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:23:39.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:23:39.586 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:39.584+0000 7fab8c540540 -1 osd.1 0 log_to_monitors true 2024-09-18T16:23:40.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:23:40.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:23:40.582 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:23:40.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:23:40.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:23:40.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:23:40.942 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/595250,v1:127.0.0.1:6811/595250] [v2:127.0.0.1:6812/595250,v1:127.0.0.1:6813/595250] exists,up e70875d3-dab0-4a0a-9227-dfaf81bd860c 2024-09-18T16:23:40.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:23:40.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:23:40.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:23:40.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:524: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:23:40.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:526: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 2 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_scrub_sleep=0.2 2024-09-18T16:23:40.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-09-18T16:23:40.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:23:40.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T16:23:40.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:23:40.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/2 2024-09-18T16:23:40.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-18T16:23:40.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:23:40.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:23:40.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:23:40.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2024-09-18T16:23:40.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2024-09-18T16:23:40.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:23:40.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:23:40.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-18T16:23:40.945 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:23:40.945 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:23:40.945 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:23:40.946 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:23:40.946 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:23:40.946 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:23:40.946 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:23:40.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:23:40.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:23:40.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:23:40.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:23:40.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-18T16:23:40.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-18T16:23:40.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:23:40.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:23:40.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:23:40.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:23:40.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:23:40.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_scrub_sleep=0.2' 2024-09-18T16:23:40.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/2 2024-09-18T16:23:40.947 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:23:40.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4b4b2031-0603-42ca-acaf-553257a7c14f 2024-09-18T16:23:40.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4b4b2031-0603-42ca-acaf-553257a7c14f' 2024-09-18T16:23:40.948 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 4b4b2031-0603-42ca-acaf-553257a7c14f 2024-09-18T16:23:40.949 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:23:40.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCM/upm2eywORAA2SJACQb0RQhmYK+cyNlpHQ== 2024-09-18T16:23:40.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCM/upm2eywORAA2SJACQb0RQhmYK+cyNlpHQ=="}' 2024-09-18T16:23:40.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4b4b2031-0603-42ca-acaf-553257a7c14f -i td/osd-scrub-test/2/new.json 2024-09-18T16:23:41.361 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:23:41.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/2/new.json 2024-09-18T16:23:41.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_scrub_sleep=0.2 --mkfs --key AQCM/upm2eywORAA2SJACQb0RQhmYK+cyNlpHQ== --osd-uuid 4b4b2031-0603-42ca-acaf-553257a7c14f 2024-09-18T16:23:41.415 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:41.412+0000 7f4525d5f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:23:41.420 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:41.416+0000 7f4525d5f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:23:41.422 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:41.420+0000 7f4525d5f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:23:41.423 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:41.420+0000 7f4525d5f540 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-09-18T16:23:41.423 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:41.420+0000 7f4525d5f540 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-09-18T16:23:41.423 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:41.420+0000 7f4525d5f540 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-09-18T16:23:41.503 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:41.500+0000 7f4525d5f540 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-09-18T16:23:47.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2024-09-18T16:23:47.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:23:47.415 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T16:23:47.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T16:23:47.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:23:47.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T16:23:47.802 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T16:23:47.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_scrub_sleep=0.2 2024-09-18T16:23:47.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:23:47.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:23:47.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:23:47.834 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:47.832+0000 7f73d3d4b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:23:47.836 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:47.832+0000 7f73d3d4b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:23:47.837 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:47.836+0000 7f73d3d4b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:23:48.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T16:23:48.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:23:48.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T16:23:48.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:23:48.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:23:48.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:23:48.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:23:48.097 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:23:48.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:23:48.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:23:48.372 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:48.368+0000 7f73d3d4b540 -1 Falling back to public interface 2024-09-18T16:23:48.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:23:49.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:23:49.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:23:49.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:23:49.446 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:23:49.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:23:49.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:23:49.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:23:50.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:23:50.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:23:50.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:23:50.753 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:23:50.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:23:50.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:23:51.040 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:23:51.036+0000 7f73d3d4b540 -1 osd.2 0 log_to_monitors true 2024-09-18T16:23:51.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:23:52.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:23:52.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:23:52.059 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:23:52.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:23:52.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:23:52.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:23:52.529 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/596649,v1:127.0.0.1:6819/596649] [v2:127.0.0.1:6820/596649,v1:127.0.0.1:6821/596649] exists,up 4b4b2031-0603-42ca-acaf-553257a7c14f 2024-09-18T16:23:52.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:23:52.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:23:52.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:23:52.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:530: TEST_dump_scrub_schedule: create_pool test 1 1 2024-09-18T16:23:52.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-18T16:23:53.049 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'test' created 2024-09-18T16:23:53.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T16:23:54.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:531: TEST_dump_scrub_schedule: wait_for_clean 2024-09-18T16:23:54.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:23:54.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:23:54.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:23:54.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:23:54.065 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:23:54.066 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:23:54.066 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:23:54.067 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:23:54.067 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:23:54.067 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:23:54.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:23:54.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:23:54.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:23:54.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=500 2024-09-18T16:23:54.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:23:54.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:23:54.589 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:23:54.589 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:23:54.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:23:54.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:23:54.589 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:23:54.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836488 2024-09-18T16:23:54.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836488 2024-09-18T16:23:54.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836488' 2024-09-18T16:23:54.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:23:54.789 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:23:54.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672964 2024-09-18T16:23:54.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672964 2024-09-18T16:23:54.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836488 1-42949672964' 2024-09-18T16:23:54.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:23:54.925 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:23:55.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509442 2024-09-18T16:23:55.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509442 2024-09-18T16:23:55.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836488 1-42949672964 2-64424509442' 2024-09-18T16:23:55.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:23:55.055 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836488 2024-09-18T16:23:55.055 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:23:55.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:23:55.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836488 2024-09-18T16:23:55.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:23:55.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836488 2024-09-18T16:23:55.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-09-18T16:23:55.059 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836488 2024-09-18T16:23:55.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:23:55.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836488 -lt 21474836488 2024-09-18T16:23:55.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:23:55.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672964 2024-09-18T16:23:55.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:23:55.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:23:55.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672964 2024-09-18T16:23:55.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:23:55.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672964 2024-09-18T16:23:55.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-09-18T16:23:55.366 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672964 2024-09-18T16:23:55.366 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:23:55.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672964 -lt 42949672964 2024-09-18T16:23:55.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:23:55.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509442 2024-09-18T16:23:55.702 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:23:55.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:23:55.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509442 2024-09-18T16:23:55.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:23:55.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509442 2024-09-18T16:23:55.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-09-18T16:23:55.708 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509442 2024-09-18T16:23:55.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:23:56.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509442 -lt 64424509442 2024-09-18T16:23:56.052 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:23:56.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:23:56.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:23:56.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:23:56.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:23:56.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:23:56.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:23:56.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:23:56.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:23:56.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:23:56.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:23:56.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:23:56.775 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:23:56.775 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:23:56.776 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:23:57.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:23:57.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:23:57.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:23:57.159 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:532: TEST_dump_scrub_schedule: ceph osd dump 2024-09-18T16:23:57.159 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:532: TEST_dump_scrub_schedule: grep '^pool.*['\'']test['\'']' 2024-09-18T16:23:57.159 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:532: TEST_dump_scrub_schedule: awk '{ print $2 }' 2024-09-18T16:23:57.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:532: TEST_dump_scrub_schedule: poolid=1 2024-09-18T16:23:57.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:534: TEST_dump_scrub_schedule: dd if=/dev/urandom of=testdata.562265 bs=1032 count=1 2024-09-18T16:23:57.504 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records in 2024-09-18T16:23:57.504 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records out 2024-09-18T16:23:57.504 INFO:tasks.workunit.client.0.smithi046.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 0.000203458 s, 5.1 MB/s 2024-09-18T16:23:57.505 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:535: TEST_dump_scrub_schedule: seq 1 15 2024-09-18T16:23:57.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:535: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-18T16:23:57.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:537: TEST_dump_scrub_schedule: rados -p test put obj1 testdata.562265 2024-09-18T16:23:57.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:535: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-18T16:23:57.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:537: TEST_dump_scrub_schedule: rados -p test put obj2 testdata.562265 2024-09-18T16:23:57.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:535: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-18T16:23:57.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:537: TEST_dump_scrub_schedule: rados -p test put obj3 testdata.562265 2024-09-18T16:23:58.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:535: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-18T16:23:58.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:537: TEST_dump_scrub_schedule: rados -p test put obj4 testdata.562265 2024-09-18T16:23:58.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:535: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-18T16:23:58.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:537: TEST_dump_scrub_schedule: rados -p test put obj5 testdata.562265 2024-09-18T16:23:58.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:535: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-18T16:23:58.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:537: TEST_dump_scrub_schedule: rados -p test put obj6 testdata.562265 2024-09-18T16:23:58.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:535: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-18T16:23:58.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:537: TEST_dump_scrub_schedule: rados -p test put obj7 testdata.562265 2024-09-18T16:23:58.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:535: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-18T16:23:58.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:537: TEST_dump_scrub_schedule: rados -p test put obj8 testdata.562265 2024-09-18T16:23:58.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:535: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-18T16:23:58.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:537: TEST_dump_scrub_schedule: rados -p test put obj9 testdata.562265 2024-09-18T16:23:58.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:535: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-18T16:23:58.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:537: TEST_dump_scrub_schedule: rados -p test put obj10 testdata.562265 2024-09-18T16:23:59.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:535: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-18T16:23:59.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:537: TEST_dump_scrub_schedule: rados -p test put obj11 testdata.562265 2024-09-18T16:23:59.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:535: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-18T16:23:59.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:537: TEST_dump_scrub_schedule: rados -p test put obj12 testdata.562265 2024-09-18T16:23:59.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:535: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-18T16:23:59.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:537: TEST_dump_scrub_schedule: rados -p test put obj13 testdata.562265 2024-09-18T16:23:59.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:535: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-18T16:23:59.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:537: TEST_dump_scrub_schedule: rados -p test put obj14 testdata.562265 2024-09-18T16:23:59.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:535: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-18T16:23:59.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:537: TEST_dump_scrub_schedule: rados -p test put obj15 testdata.562265 2024-09-18T16:23:59.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:539: TEST_dump_scrub_schedule: rm -f testdata.562265 2024-09-18T16:23:59.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:541: TEST_dump_scrub_schedule: local pgid=1.0 2024-09-18T16:23:59.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:542: TEST_dump_scrub_schedule: date -Ins 2024-09-18T16:23:59.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:542: TEST_dump_scrub_schedule: local now_is=2024-09-18T16:23:59,832805039+00:00 2024-09-18T16:23:59.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:550: TEST_dump_scrub_schedule: expct_starting=(['query_active']="false" ['query_is_future']="true" ['query_schedule']="scrub scheduled") 2024-09-18T16:23:59.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:550: TEST_dump_scrub_schedule: declare -A expct_starting 2024-09-18T16:23:59.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:551: TEST_dump_scrub_schedule: declare -A sched_data 2024-09-18T16:23:59.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:552: TEST_dump_scrub_schedule: extract_published_sch 1.0 2024-09-18T16:23:59,832805039+00:00 2019-10-12T20:32:43.645168+0000 sched_data 2024-09-18T16:23:59.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.0 2024-09-18T16:23:59.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sched_data 2024-09-18T16:23:59.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2024-09-18T16:23:59,832805039+00:00 2024-09-18T16:23:59.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2019-10-12T20:32:43.645168+0000 2024-09-18T16:23:59.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=1 2024-09-18T16:23:59.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2024-09-18T16:23:59.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2024-09-18T16:24:00.162 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:24:00.349 INFO:tasks.workunit.client.0.smithi046.stdout:( 2024-09-18T16:24:00.349 INFO:tasks.workunit.client.0.smithi046.stdout:[query_epoch]=19 2024-09-18T16:24:00.349 INFO:tasks.workunit.client.0.smithi046.stdout:[query_seq]=26 2024-09-18T16:24:00.349 INFO:tasks.workunit.client.0.smithi046.stdout:[query_active]=false 2024-09-18T16:24:00.349 INFO:tasks.workunit.client.0.smithi046.stdout:[query_schedule]='scrub scheduled' 2024-09-18T16:24:00.349 INFO:tasks.workunit.client.0.smithi046.stdout:[query_schedule_at]='2024-09-19T16:23:52.902209+0000' 2024-09-18T16:24:00.349 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_duration]=0 2024-09-18T16:24:00.349 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_stamp]='2024-09-18T16:23:52.902209+0000' 2024-09-18T16:24:00.350 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_scrub]='0x0' 2024-09-18T16:24:00.350 INFO:tasks.workunit.client.0.smithi046.stdout:[query_is_future]=true 2024-09-18T16:24:00.350 INFO:tasks.workunit.client.0.smithi046.stdout:[query_vs_date]=true 2024-09-18T16:24:00.350 INFO:tasks.workunit.client.0.smithi046.stdout:[query_scrub_seq]=null 2024-09-18T16:24:00.350 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_pg_state]='active+clean' 2024-09-18T16:24:00.350 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_state_has_scrubbing]=false 2024-09-18T16:24:00.350 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_last_duration]=0 2024-09-18T16:24:00.350 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-18T16:24:00.350 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_schedule_at]='2024-09-19T16:23:52.902209+0000' 2024-09-18T16:24:00.350 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_is_future]=true 2024-09-18T16:24:00.350 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_vs_date]=true 2024-09-18T16:24:00.350 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_reported_epoch]=18 2024-09-18T16:24:00.350 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_seq]=9 2024-09-18T16:24:00.351 INFO:tasks.workunit.client.0.smithi046.stdout:) 2024-09-18T16:24:00.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:553: TEST_dump_scrub_schedule: schedule_against_expected sched_data expct_starting initial 2024-09-18T16:24:00.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:170: schedule_against_expected: local -n dict=sched_data 2024-09-18T16:24:00.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:171: schedule_against_expected: local -n ep=expct_starting 2024-09-18T16:24:00.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:172: schedule_against_expected: local extr_dbg=1 2024-09-18T16:24:00.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:175: schedule_against_expected: local saved_echo_flag=x 2024-09-18T16:24:00.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:176: schedule_against_expected: set +x 2024-09-18T16:24:00.398 INFO:tasks.workunit.client.0.smithi046.stdout:-- - comparing: 2024-09-18T16:24:00.398 INFO:tasks.workunit.client.0.smithi046.stdout:key is query_is_future expected: true in actual: true 2024-09-18T16:24:00.398 INFO:tasks.workunit.client.0.smithi046.stdout:key is query_schedule expected: scrub scheduled in actual: scrub scheduled 2024-09-18T16:24:00.398 INFO:tasks.workunit.client.0.smithi046.stdout:key is query_active expected: false in actual: false 2024-09-18T16:24:00.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:195: schedule_against_expected: return 0 2024-09-18T16:24:00.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:554: TEST_dump_scrub_schedule: (( 0 == 0 )) 2024-09-18T16:24:00.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:555: TEST_dump_scrub_schedule: echo 'last-scrub --- ' 0x0 2024-09-18T16:24:00.399 INFO:tasks.workunit.client.0.smithi046.stdout:last-scrub --- 0x0 2024-09-18T16:24:00.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:561: TEST_dump_scrub_schedule: saved_last_stamp=2024-09-18T16:23:52.902209+0000 2024-09-18T16:24:00.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_scrub_sleep 0 2024-09-18T16:24:00.522 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0: { 2024-09-18T16:24:00.522 INFO:tasks.workunit.client.0.smithi046.stdout: "success": "osd_scrub_sleep = '' " 2024-09-18T16:24:00.523 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:24:00.536 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1: { 2024-09-18T16:24:00.536 INFO:tasks.workunit.client.0.smithi046.stdout: "success": "osd_scrub_sleep = '' " 2024-09-18T16:24:00.536 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:24:00.551 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2: { 2024-09-18T16:24:00.551 INFO:tasks.workunit.client.0.smithi046.stdout: "success": "osd_scrub_sleep = '' " 2024-09-18T16:24:00.551 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:24:00.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:563: TEST_dump_scrub_schedule: ceph pg deep-scrub 1.0 2024-09-18T16:24:00.880 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 1.0 on osd.1 to deep-scrub 2024-09-18T16:24:00.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:564: TEST_dump_scrub_schedule: ceph pg scrub 1.0 2024-09-18T16:24:01.190 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 1.0 on osd.1 to scrub 2024-09-18T16:24:01.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:569: TEST_dump_scrub_schedule: sleep 5 2024-09-18T16:24:06.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:570: TEST_dump_scrub_schedule: sched_data=() 2024-09-18T16:24:06.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:571: TEST_dump_scrub_schedule: expct_qry_duration=(['query_last_duration']="0" ['query_last_duration_neg']="not0") 2024-09-18T16:24:06.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:571: TEST_dump_scrub_schedule: declare -A expct_qry_duration 2024-09-18T16:24:06.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:572: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-09-18T16:23:52.902209+0000 expct_qry_duration 'WaitingAfterScrub ' sched_data 2024-09-18T16:24:06.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:98: wait_any_cond: local pgid=1.0 2024-09-18T16:24:06.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:99: wait_any_cond: local retries=10 2024-09-18T16:24:06.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:100: wait_any_cond: local cmp_date=2024-09-18T16:23:52.902209+0000 2024-09-18T16:24:06.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:101: wait_any_cond: local -n ep=expct_qry_duration 2024-09-18T16:24:06.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:102: wait_any_cond: local -n out_array=sched_data 2024-09-18T16:24:06.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:103: wait_any_cond: local -A sc_data 2024-09-18T16:24:06.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:104: wait_any_cond: local extr_dbg=2 2024-09-18T16:24:06.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local saved_echo_flag=x 2024-09-18T16:24:06.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: set +x 2024-09-18T16:24:06.207 INFO:tasks.workunit.client.0.smithi046.stdout:waiting for any condition (WaitingAfterScrub ): pg:1.0 dt:2024-09-18T16:23:52.902209+0000 (10 retries) 2024-09-18T16:24:07.012 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:24:07.234 INFO:tasks.workunit.client.0.smithi046.stdout:( 2024-09-18T16:24:07.234 INFO:tasks.workunit.client.0.smithi046.stdout:[query_epoch]=19 2024-09-18T16:24:07.234 INFO:tasks.workunit.client.0.smithi046.stdout:[query_seq]=38 2024-09-18T16:24:07.234 INFO:tasks.workunit.client.0.smithi046.stdout:[query_active]=false 2024-09-18T16:24:07.234 INFO:tasks.workunit.client.0.smithi046.stdout:[query_schedule]='scrub scheduled' 2024-09-18T16:24:07.234 INFO:tasks.workunit.client.0.smithi046.stdout:[query_schedule_at]='2024-09-19T16:24:01.528947+0000' 2024-09-18T16:24:07.235 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_duration]=1 2024-09-18T16:24:07.235 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_stamp]='2024-09-18T16:24:01.528947+0000' 2024-09-18T16:24:07.235 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_scrub]='19x15' 2024-09-18T16:24:07.235 INFO:tasks.workunit.client.0.smithi046.stdout:[query_is_future]=true 2024-09-18T16:24:07.235 INFO:tasks.workunit.client.0.smithi046.stdout:[query_vs_date]=true 2024-09-18T16:24:07.235 INFO:tasks.workunit.client.0.smithi046.stdout:[query_scrub_seq]=null 2024-09-18T16:24:07.235 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_pg_state]='active+clean' 2024-09-18T16:24:07.235 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_state_has_scrubbing]=false 2024-09-18T16:24:07.235 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_last_duration]=0 2024-09-18T16:24:07.235 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_schedule]='queued for deep scrub' 2024-09-18T16:24:07.235 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_schedule_at]='0' 2024-09-18T16:24:07.235 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_is_future]=false 2024-09-18T16:24:07.235 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_vs_date]=false 2024-09-18T16:24:07.235 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_reported_epoch]=19 2024-09-18T16:24:07.236 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_seq]=26 2024-09-18T16:24:07.236 INFO:tasks.workunit.client.0.smithi046.stdout:) 2024-09-18T16:24:07.280 INFO:tasks.workunit.client.0.smithi046.stdout:--> loop: 1 ~ false / 38 / 26 / true / 2024-09-18T16:24:01.528947+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-09-18T16:24:07.280 INFO:tasks.workunit.client.0.smithi046.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 1 2024-09-18T16:24:07.280 INFO:tasks.workunit.client.0.smithi046.stdout:WaitingAfterScrub - 'query_last_duration' actual value (1) matches expected (0) (negation: 1) 2024-09-18T16:24:07.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:149: wait_any_cond: return 0 2024-09-18T16:24:07.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:574: TEST_dump_scrub_schedule: sched_data=() 2024-09-18T16:24:07.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:575: TEST_dump_scrub_schedule: expct_dmp_duration=(['dmp_last_duration']="0" ['dmp_last_duration_neg']="not0") 2024-09-18T16:24:07.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:575: TEST_dump_scrub_schedule: declare -A expct_dmp_duration 2024-09-18T16:24:07.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:576: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-09-18T16:23:52.902209+0000 expct_dmp_duration 'WaitingAfterScrub_dmp ' sched_data 2024-09-18T16:24:07.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:98: wait_any_cond: local pgid=1.0 2024-09-18T16:24:07.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:99: wait_any_cond: local retries=10 2024-09-18T16:24:07.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:100: wait_any_cond: local cmp_date=2024-09-18T16:23:52.902209+0000 2024-09-18T16:24:07.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:101: wait_any_cond: local -n ep=expct_dmp_duration 2024-09-18T16:24:07.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:102: wait_any_cond: local -n out_array=sched_data 2024-09-18T16:24:07.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:103: wait_any_cond: local -A sc_data 2024-09-18T16:24:07.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:104: wait_any_cond: local extr_dbg=2 2024-09-18T16:24:07.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local saved_echo_flag=x 2024-09-18T16:24:07.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: set +x 2024-09-18T16:24:07.285 INFO:tasks.workunit.client.0.smithi046.stdout:waiting for any condition (WaitingAfterScrub_dmp ): pg:1.0 dt:2024-09-18T16:23:52.902209+0000 (10 retries) 2024-09-18T16:24:08.095 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:24:08.285 INFO:tasks.workunit.client.0.smithi046.stdout:( 2024-09-18T16:24:08.285 INFO:tasks.workunit.client.0.smithi046.stdout:[query_epoch]=19 2024-09-18T16:24:08.285 INFO:tasks.workunit.client.0.smithi046.stdout:[query_seq]=38 2024-09-18T16:24:08.285 INFO:tasks.workunit.client.0.smithi046.stdout:[query_active]=false 2024-09-18T16:24:08.285 INFO:tasks.workunit.client.0.smithi046.stdout:[query_schedule]='scrub scheduled' 2024-09-18T16:24:08.285 INFO:tasks.workunit.client.0.smithi046.stdout:[query_schedule_at]='2024-09-19T16:24:01.528947+0000' 2024-09-18T16:24:08.285 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_duration]=1 2024-09-18T16:24:08.285 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_stamp]='2024-09-18T16:24:01.528947+0000' 2024-09-18T16:24:08.285 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_scrub]='19x15' 2024-09-18T16:24:08.286 INFO:tasks.workunit.client.0.smithi046.stdout:[query_is_future]=true 2024-09-18T16:24:08.286 INFO:tasks.workunit.client.0.smithi046.stdout:[query_vs_date]=true 2024-09-18T16:24:08.286 INFO:tasks.workunit.client.0.smithi046.stdout:[query_scrub_seq]=null 2024-09-18T16:24:08.286 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_pg_state]='active+clean' 2024-09-18T16:24:08.286 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_state_has_scrubbing]=false 2024-09-18T16:24:08.286 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_last_duration]=1 2024-09-18T16:24:08.286 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-18T16:24:08.286 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_schedule_at]='2024-09-19T16:24:01.528947+0000' 2024-09-18T16:24:08.286 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_is_future]=true 2024-09-18T16:24:08.286 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_vs_date]=true 2024-09-18T16:24:08.286 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_reported_epoch]=19 2024-09-18T16:24:08.286 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_seq]=37 2024-09-18T16:24:08.286 INFO:tasks.workunit.client.0.smithi046.stdout:) 2024-09-18T16:24:08.333 INFO:tasks.workunit.client.0.smithi046.stdout:--> loop: 1 ~ false / 38 / 37 / true / 2024-09-18T16:24:01.528947+0000 / scrub scheduled %%% dmp_last_duration_neg dmp_last_duration 2024-09-18T16:24:08.333 INFO:tasks.workunit.client.0.smithi046.stdout:key is dmp_last_duration: negation:1 # expected: 0 # in actual: 1 2024-09-18T16:24:08.333 INFO:tasks.workunit.client.0.smithi046.stdout:WaitingAfterScrub_dmp - 'dmp_last_duration' actual value (1) matches expected (0) (negation: 1) 2024-09-18T16:24:08.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:149: wait_any_cond: return 0 2024-09-18T16:24:08.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:578: TEST_dump_scrub_schedule: sleep 2 2024-09-18T16:24:10.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:585: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_scrub_chunk_max 3 2024-09-18T16:24:10.472 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0: { 2024-09-18T16:24:10.472 INFO:tasks.workunit.client.0.smithi046.stdout: "success": "osd_scrub_chunk_max = '' (not observed, change may require restart) " 2024-09-18T16:24:10.472 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:24:10.487 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1: { 2024-09-18T16:24:10.487 INFO:tasks.workunit.client.0.smithi046.stdout: "success": "osd_scrub_chunk_max = '' (not observed, change may require restart) " 2024-09-18T16:24:10.487 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:24:10.503 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2: { 2024-09-18T16:24:10.503 INFO:tasks.workunit.client.0.smithi046.stdout: "success": "osd_scrub_chunk_max = '' (not observed, change may require restart) " 2024-09-18T16:24:10.503 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:24:10.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:586: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_scrub_sleep 1.0 2024-09-18T16:24:10.645 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0: { 2024-09-18T16:24:10.645 INFO:tasks.workunit.client.0.smithi046.stdout: "success": "osd_scrub_sleep = '' " 2024-09-18T16:24:10.645 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:24:10.659 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1: { 2024-09-18T16:24:10.659 INFO:tasks.workunit.client.0.smithi046.stdout: "success": "osd_scrub_sleep = '' " 2024-09-18T16:24:10.659 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:24:10.674 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2: { 2024-09-18T16:24:10.674 INFO:tasks.workunit.client.0.smithi046.stdout: "success": "osd_scrub_sleep = '' " 2024-09-18T16:24:10.674 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:24:10.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:587: TEST_dump_scrub_schedule: ceph osd set noscrub 2024-09-18T16:24:11.119 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T16:24:11.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:588: TEST_dump_scrub_schedule: sleep 2 2024-09-18T16:24:13.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:589: TEST_dump_scrub_schedule: saved_last_stamp=2024-09-18T16:24:01.528947+0000 2024-09-18T16:24:13.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:591: TEST_dump_scrub_schedule: ceph pg 1.0 scrub 2024-09-18T16:24:13.261 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:24:13.261 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T16:24:13.262 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T16:22:33.259759+0000" 2024-09-18T16:24:13.262 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:24:13.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:592: TEST_dump_scrub_schedule: sleep 1 2024-09-18T16:24:14.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:593: TEST_dump_scrub_schedule: sched_data=() 2024-09-18T16:24:14.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:594: TEST_dump_scrub_schedule: expct_scrub_peri_sched=(['query_is_future']="false") 2024-09-18T16:24:14.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:594: TEST_dump_scrub_schedule: declare -A expct_scrub_peri_sched 2024-09-18T16:24:14.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:595: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-09-18T16:24:01.528947+0000 expct_scrub_peri_sched waitingBeingScheduled sched_data 2024-09-18T16:24:14.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:98: wait_any_cond: local pgid=1.0 2024-09-18T16:24:14.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:99: wait_any_cond: local retries=10 2024-09-18T16:24:14.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:100: wait_any_cond: local cmp_date=2024-09-18T16:24:01.528947+0000 2024-09-18T16:24:14.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:101: wait_any_cond: local -n ep=expct_scrub_peri_sched 2024-09-18T16:24:14.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:102: wait_any_cond: local -n out_array=sched_data 2024-09-18T16:24:14.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:103: wait_any_cond: local -A sc_data 2024-09-18T16:24:14.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:104: wait_any_cond: local extr_dbg=2 2024-09-18T16:24:14.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local saved_echo_flag=x 2024-09-18T16:24:14.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: set +x 2024-09-18T16:24:14.277 INFO:tasks.workunit.client.0.smithi046.stdout:waiting for any condition (waitingBeingScheduled): pg:1.0 dt:2024-09-18T16:24:01.528947+0000 (10 retries) 2024-09-18T16:24:15.078 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:24:15.265 INFO:tasks.workunit.client.0.smithi046.stdout:( 2024-09-18T16:24:15.265 INFO:tasks.workunit.client.0.smithi046.stdout:[query_epoch]=20 2024-09-18T16:24:15.265 INFO:tasks.workunit.client.0.smithi046.stdout:[query_seq]=40 2024-09-18T16:24:15.265 INFO:tasks.workunit.client.0.smithi046.stdout:[query_active]=false 2024-09-18T16:24:15.265 INFO:tasks.workunit.client.0.smithi046.stdout:[query_schedule]='queued for scrub' 2024-09-18T16:24:15.265 INFO:tasks.workunit.client.0.smithi046.stdout:[query_schedule_at]='0' 2024-09-18T16:24:15.265 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_duration]=1 2024-09-18T16:24:15.266 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_stamp]='2024-09-11T16:22:33.259759+0000' 2024-09-18T16:24:15.266 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_scrub]='19x15' 2024-09-18T16:24:15.266 INFO:tasks.workunit.client.0.smithi046.stdout:[query_is_future]=false 2024-09-18T16:24:15.266 INFO:tasks.workunit.client.0.smithi046.stdout:[query_vs_date]=false 2024-09-18T16:24:15.266 INFO:tasks.workunit.client.0.smithi046.stdout:[query_scrub_seq]=null 2024-09-18T16:24:15.266 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_pg_state]='active+clean' 2024-09-18T16:24:15.266 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_state_has_scrubbing]=false 2024-09-18T16:24:15.266 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_last_duration]=1 2024-09-18T16:24:15.266 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-18T16:24:15.266 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_schedule_at]='2024-09-19T16:24:01.528947+0000' 2024-09-18T16:24:15.266 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_is_future]=true 2024-09-18T16:24:15.266 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_vs_date]=true 2024-09-18T16:24:15.266 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_reported_epoch]=19 2024-09-18T16:24:15.266 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_seq]=37 2024-09-18T16:24:15.266 INFO:tasks.workunit.client.0.smithi046.stdout:) 2024-09-18T16:24:15.312 INFO:tasks.workunit.client.0.smithi046.stdout:--> loop: 1 ~ false / 40 / 37 / false / 2024-09-11T16:22:33.259759+0000 / queued for scrub %%% query_is_future 2024-09-18T16:24:15.312 INFO:tasks.workunit.client.0.smithi046.stdout:key is query_is_future: negation:0 # expected: false # in actual: false 2024-09-18T16:24:15.312 INFO:tasks.workunit.client.0.smithi046.stdout:waitingBeingScheduled - 'query_is_future' actual value (false) matches expected (false) (negation: 0) 2024-09-18T16:24:15.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:149: wait_any_cond: return 0 2024-09-18T16:24:15.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:607: TEST_dump_scrub_schedule: saved_last_stamp=2024-09-11T16:22:33.259759+0000 2024-09-18T16:24:15.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:608: TEST_dump_scrub_schedule: ceph osd unset noscrub 2024-09-18T16:24:15.737 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is unset 2024-09-18T16:24:15.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:610: TEST_dump_scrub_schedule: cond_active=(['query_active']="true") 2024-09-18T16:24:15.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:610: TEST_dump_scrub_schedule: declare -A cond_active 2024-09-18T16:24:15.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:611: TEST_dump_scrub_schedule: sched_data=() 2024-09-18T16:24:15.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:612: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-09-11T16:22:33.259759+0000 cond_active 'WaitingActive ' sched_data 2024-09-18T16:24:15.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:98: wait_any_cond: local pgid=1.0 2024-09-18T16:24:15.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:99: wait_any_cond: local retries=10 2024-09-18T16:24:15.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:100: wait_any_cond: local cmp_date=2024-09-11T16:22:33.259759+0000 2024-09-18T16:24:15.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:101: wait_any_cond: local -n ep=cond_active 2024-09-18T16:24:15.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:102: wait_any_cond: local -n out_array=sched_data 2024-09-18T16:24:15.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:103: wait_any_cond: local -A sc_data 2024-09-18T16:24:15.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:104: wait_any_cond: local extr_dbg=2 2024-09-18T16:24:15.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local saved_echo_flag=x 2024-09-18T16:24:15.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: set +x 2024-09-18T16:24:15.752 INFO:tasks.workunit.client.0.smithi046.stdout:waiting for any condition (WaitingActive ): pg:1.0 dt:2024-09-11T16:22:33.259759+0000 (10 retries) 2024-09-18T16:24:16.550 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:24:16.736 INFO:tasks.workunit.client.0.smithi046.stdout:( 2024-09-18T16:24:16.736 INFO:tasks.workunit.client.0.smithi046.stdout:[query_epoch]=20 2024-09-18T16:24:16.737 INFO:tasks.workunit.client.0.smithi046.stdout:[query_seq]=40 2024-09-18T16:24:16.737 INFO:tasks.workunit.client.0.smithi046.stdout:[query_active]=false 2024-09-18T16:24:16.737 INFO:tasks.workunit.client.0.smithi046.stdout:[query_schedule]='queued for scrub' 2024-09-18T16:24:16.737 INFO:tasks.workunit.client.0.smithi046.stdout:[query_schedule_at]='0' 2024-09-18T16:24:16.737 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_duration]=1 2024-09-18T16:24:16.737 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_stamp]='2024-09-11T16:22:33.259759+0000' 2024-09-18T16:24:16.737 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_scrub]='19x15' 2024-09-18T16:24:16.737 INFO:tasks.workunit.client.0.smithi046.stdout:[query_is_future]=false 2024-09-18T16:24:16.737 INFO:tasks.workunit.client.0.smithi046.stdout:[query_vs_date]=false 2024-09-18T16:24:16.737 INFO:tasks.workunit.client.0.smithi046.stdout:[query_scrub_seq]=null 2024-09-18T16:24:16.737 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_pg_state]='active+clean' 2024-09-18T16:24:16.737 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_state_has_scrubbing]=false 2024-09-18T16:24:16.738 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_last_duration]=1 2024-09-18T16:24:16.738 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-18T16:24:16.738 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_schedule_at]='2024-09-19T16:24:01.528947+0000' 2024-09-18T16:24:16.738 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_is_future]=true 2024-09-18T16:24:16.738 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_vs_date]=true 2024-09-18T16:24:16.738 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_reported_epoch]=19 2024-09-18T16:24:16.738 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_seq]=37 2024-09-18T16:24:16.738 INFO:tasks.workunit.client.0.smithi046.stdout:) 2024-09-18T16:24:16.784 INFO:tasks.workunit.client.0.smithi046.stdout:--> loop: 1 ~ false / 40 / 37 / false / 2024-09-11T16:22:33.259759+0000 / queued for scrub %%% query_active 2024-09-18T16:24:16.784 INFO:tasks.workunit.client.0.smithi046.stdout:key is query_active: negation:0 # expected: true # in actual: false 2024-09-18T16:24:17.583 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:24:17.814 INFO:tasks.workunit.client.0.smithi046.stdout:( 2024-09-18T16:24:17.814 INFO:tasks.workunit.client.0.smithi046.stdout:[query_epoch]=21 2024-09-18T16:24:17.814 INFO:tasks.workunit.client.0.smithi046.stdout:[query_seq]=45 2024-09-18T16:24:17.814 INFO:tasks.workunit.client.0.smithi046.stdout:[query_active]=true 2024-09-18T16:24:17.814 INFO:tasks.workunit.client.0.smithi046.stdout:[query_schedule]='scrubbing' 2024-09-18T16:24:17.815 INFO:tasks.workunit.client.0.smithi046.stdout:[query_schedule_at]='0' 2024-09-18T16:24:17.815 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_duration]=1 2024-09-18T16:24:17.815 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_stamp]='2024-09-11T16:22:33.259759+0000' 2024-09-18T16:24:17.815 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_scrub]='19x15' 2024-09-18T16:24:17.815 INFO:tasks.workunit.client.0.smithi046.stdout:[query_is_future]=false 2024-09-18T16:24:17.815 INFO:tasks.workunit.client.0.smithi046.stdout:[query_vs_date]=false 2024-09-18T16:24:17.815 INFO:tasks.workunit.client.0.smithi046.stdout:[query_scrub_seq]=null 2024-09-18T16:24:17.815 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_pg_state]='active+clean' 2024-09-18T16:24:17.815 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_state_has_scrubbing]=false 2024-09-18T16:24:17.815 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_last_duration]=1 2024-09-18T16:24:17.815 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-18T16:24:17.815 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_schedule_at]='2024-09-19T16:24:01.528947+0000' 2024-09-18T16:24:17.815 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_is_future]=true 2024-09-18T16:24:17.816 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_vs_date]=true 2024-09-18T16:24:17.816 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_reported_epoch]=20 2024-09-18T16:24:17.816 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_seq]=39 2024-09-18T16:24:17.816 INFO:tasks.workunit.client.0.smithi046.stdout:) 2024-09-18T16:24:17.860 INFO:tasks.workunit.client.0.smithi046.stdout:--> loop: 2 ~ true / 45 / 39 / false / 2024-09-11T16:22:33.259759+0000 / scrubbing %%% query_active 2024-09-18T16:24:17.860 INFO:tasks.workunit.client.0.smithi046.stdout:key is query_active: negation:0 # expected: true # in actual: true 2024-09-18T16:24:17.860 INFO:tasks.workunit.client.0.smithi046.stdout:WaitingActive - 'query_active' actual value (true) matches expected (true) (negation: 0) 2024-09-18T16:24:17.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:149: wait_any_cond: return 0 2024-09-18T16:24:17.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:616: TEST_dump_scrub_schedule: cond_active_dmp=(['dmp_state_has_scrubbing']="true" ['query_active']="false") 2024-09-18T16:24:17.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:616: TEST_dump_scrub_schedule: declare -A cond_active_dmp 2024-09-18T16:24:17.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:617: TEST_dump_scrub_schedule: sched_data=() 2024-09-18T16:24:17.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:618: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-09-11T16:22:33.259759+0000 cond_active_dmp 'WaitingActive ' sched_data 2024-09-18T16:24:17.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:98: wait_any_cond: local pgid=1.0 2024-09-18T16:24:17.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:99: wait_any_cond: local retries=10 2024-09-18T16:24:17.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:100: wait_any_cond: local cmp_date=2024-09-11T16:22:33.259759+0000 2024-09-18T16:24:17.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:101: wait_any_cond: local -n ep=cond_active_dmp 2024-09-18T16:24:17.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:102: wait_any_cond: local -n out_array=sched_data 2024-09-18T16:24:17.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:103: wait_any_cond: local -A sc_data 2024-09-18T16:24:17.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:104: wait_any_cond: local extr_dbg=2 2024-09-18T16:24:17.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local saved_echo_flag=x 2024-09-18T16:24:17.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: set +x 2024-09-18T16:24:17.863 INFO:tasks.workunit.client.0.smithi046.stdout:waiting for any condition (WaitingActive ): pg:1.0 dt:2024-09-11T16:22:33.259759+0000 (10 retries) 2024-09-18T16:24:18.666 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:24:18.855 INFO:tasks.workunit.client.0.smithi046.stdout:( 2024-09-18T16:24:18.855 INFO:tasks.workunit.client.0.smithi046.stdout:[query_epoch]=21 2024-09-18T16:24:18.855 INFO:tasks.workunit.client.0.smithi046.stdout:[query_seq]=46 2024-09-18T16:24:18.855 INFO:tasks.workunit.client.0.smithi046.stdout:[query_active]=true 2024-09-18T16:24:18.856 INFO:tasks.workunit.client.0.smithi046.stdout:[query_schedule]='scrubbing' 2024-09-18T16:24:18.856 INFO:tasks.workunit.client.0.smithi046.stdout:[query_schedule_at]='0' 2024-09-18T16:24:18.856 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_duration]=1 2024-09-18T16:24:18.856 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_stamp]='2024-09-11T16:22:33.259759+0000' 2024-09-18T16:24:18.856 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_scrub]='19x15' 2024-09-18T16:24:18.856 INFO:tasks.workunit.client.0.smithi046.stdout:[query_is_future]=false 2024-09-18T16:24:18.856 INFO:tasks.workunit.client.0.smithi046.stdout:[query_vs_date]=false 2024-09-18T16:24:18.856 INFO:tasks.workunit.client.0.smithi046.stdout:[query_scrub_seq]=null 2024-09-18T16:24:18.856 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_pg_state]='active+clean' 2024-09-18T16:24:18.856 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_state_has_scrubbing]=false 2024-09-18T16:24:18.856 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_last_duration]=1 2024-09-18T16:24:18.856 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-18T16:24:18.857 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_schedule_at]='2024-09-19T16:24:01.528947+0000' 2024-09-18T16:24:18.857 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_is_future]=true 2024-09-18T16:24:18.857 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_vs_date]=true 2024-09-18T16:24:18.857 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_reported_epoch]=20 2024-09-18T16:24:18.857 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_seq]=39 2024-09-18T16:24:18.857 INFO:tasks.workunit.client.0.smithi046.stdout:) 2024-09-18T16:24:18.905 INFO:tasks.workunit.client.0.smithi046.stdout:--> loop: 1 ~ true / 46 / 39 / false / 2024-09-11T16:22:33.259759+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2024-09-18T16:24:18.905 INFO:tasks.workunit.client.0.smithi046.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-09-18T16:24:18.905 INFO:tasks.workunit.client.0.smithi046.stdout:key is query_active: negation:0 # expected: false # in actual: true 2024-09-18T16:24:19.701 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:24:19.887 INFO:tasks.workunit.client.0.smithi046.stdout:( 2024-09-18T16:24:19.887 INFO:tasks.workunit.client.0.smithi046.stdout:[query_epoch]=21 2024-09-18T16:24:19.888 INFO:tasks.workunit.client.0.smithi046.stdout:[query_seq]=47 2024-09-18T16:24:19.888 INFO:tasks.workunit.client.0.smithi046.stdout:[query_active]=true 2024-09-18T16:24:19.888 INFO:tasks.workunit.client.0.smithi046.stdout:[query_schedule]='scrubbing' 2024-09-18T16:24:19.888 INFO:tasks.workunit.client.0.smithi046.stdout:[query_schedule_at]='0' 2024-09-18T16:24:19.888 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_duration]=1 2024-09-18T16:24:19.888 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_stamp]='2024-09-11T16:22:33.259759+0000' 2024-09-18T16:24:19.888 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_scrub]='19x15' 2024-09-18T16:24:19.888 INFO:tasks.workunit.client.0.smithi046.stdout:[query_is_future]=false 2024-09-18T16:24:19.888 INFO:tasks.workunit.client.0.smithi046.stdout:[query_vs_date]=false 2024-09-18T16:24:19.888 INFO:tasks.workunit.client.0.smithi046.stdout:[query_scrub_seq]=null 2024-09-18T16:24:19.888 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_pg_state]='active+clean' 2024-09-18T16:24:19.888 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_state_has_scrubbing]=false 2024-09-18T16:24:19.889 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_last_duration]=1 2024-09-18T16:24:19.889 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-18T16:24:19.889 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_schedule_at]='2024-09-19T16:24:01.528947+0000' 2024-09-18T16:24:19.889 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_is_future]=true 2024-09-18T16:24:19.889 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_vs_date]=true 2024-09-18T16:24:19.889 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_reported_epoch]=20 2024-09-18T16:24:19.889 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_seq]=39 2024-09-18T16:24:19.889 INFO:tasks.workunit.client.0.smithi046.stdout:) 2024-09-18T16:24:19.932 INFO:tasks.workunit.client.0.smithi046.stdout:--> loop: 2 ~ true / 47 / 39 / false / 2024-09-11T16:22:33.259759+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2024-09-18T16:24:19.932 INFO:tasks.workunit.client.0.smithi046.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-09-18T16:24:19.932 INFO:tasks.workunit.client.0.smithi046.stdout:key is query_active: negation:0 # expected: false # in actual: true 2024-09-18T16:24:20.736 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:24:20.925 INFO:tasks.workunit.client.0.smithi046.stdout:( 2024-09-18T16:24:20.925 INFO:tasks.workunit.client.0.smithi046.stdout:[query_epoch]=21 2024-09-18T16:24:20.926 INFO:tasks.workunit.client.0.smithi046.stdout:[query_seq]=53 2024-09-18T16:24:20.926 INFO:tasks.workunit.client.0.smithi046.stdout:[query_active]=false 2024-09-18T16:24:20.926 INFO:tasks.workunit.client.0.smithi046.stdout:[query_schedule]='scrub scheduled' 2024-09-18T16:24:20.926 INFO:tasks.workunit.client.0.smithi046.stdout:[query_schedule_at]='2024-09-19T16:24:20.587345+0000' 2024-09-18T16:24:20.926 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_duration]=4 2024-09-18T16:24:20.926 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_stamp]='2024-09-18T16:24:20.587345+0000' 2024-09-18T16:24:20.926 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_scrub]='19x15' 2024-09-18T16:24:20.926 INFO:tasks.workunit.client.0.smithi046.stdout:[query_is_future]=true 2024-09-18T16:24:20.926 INFO:tasks.workunit.client.0.smithi046.stdout:[query_vs_date]=true 2024-09-18T16:24:20.926 INFO:tasks.workunit.client.0.smithi046.stdout:[query_scrub_seq]=null 2024-09-18T16:24:20.926 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_pg_state]='active+clean' 2024-09-18T16:24:20.926 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_state_has_scrubbing]=false 2024-09-18T16:24:20.926 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_last_duration]=1 2024-09-18T16:24:20.926 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-18T16:24:20.927 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_schedule_at]='2024-09-19T16:24:01.528947+0000' 2024-09-18T16:24:20.927 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_is_future]=true 2024-09-18T16:24:20.927 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_vs_date]=true 2024-09-18T16:24:20.927 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_reported_epoch]=20 2024-09-18T16:24:20.927 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_seq]=39 2024-09-18T16:24:20.927 INFO:tasks.workunit.client.0.smithi046.stdout:) 2024-09-18T16:24:20.970 INFO:tasks.workunit.client.0.smithi046.stdout:--> loop: 3 ~ false / 53 / 39 / true / 2024-09-18T16:24:20.587345+0000 / scrub scheduled %%% dmp_state_has_scrubbing query_active 2024-09-18T16:24:20.971 INFO:tasks.workunit.client.0.smithi046.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-09-18T16:24:20.971 INFO:tasks.workunit.client.0.smithi046.stdout:key is query_active: negation:0 # expected: false # in actual: false 2024-09-18T16:24:20.971 INFO:tasks.workunit.client.0.smithi046.stdout:WaitingActive - 'query_active' actual value (false) matches expected (false) (negation: 0) 2024-09-18T16:24:20.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:149: wait_any_cond: return 0 2024-09-18T16:24:20.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: teardown td/osd-scrub-test 2024-09-18T16:24:20.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-09-18T16:24:20.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:24:20.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-18T16:24:20.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:24:20.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:24:20.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:24:20.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:24:20.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:24:21.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:24:21.088 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:24:21.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:24:21.090 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:24:21.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:24:21.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:24:21.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:24:21.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:24:21.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:24:21.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:24:21.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:24:21.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:24:21.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:24:21.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-09-18T16:24:21.111 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:24:21.111 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:24:21.111 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:24:21.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.562265 2024-09-18T16:24:21.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:24:21.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:24:21.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-09-18T16:24:21.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: setup td/osd-scrub-test 2024-09-18T16:24:21.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-test 2024-09-18T16:24:21.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-test 2024-09-18T16:24:21.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-09-18T16:24:21.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:24:21.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-18T16:24:21.115 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:24:21.115 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:24:21.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:24:21.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:24:21.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:24:21.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:24:21.120 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:24:21.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:24:21.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:24:21.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:24:21.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:24:21.124 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:24:21.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:24:21.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:24:21.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:24:21.127 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:24:21.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:24:21.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:24:21.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-09-18T16:24:21.133 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:24:21.133 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:24:21.134 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:24:21.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.562265 2024-09-18T16:24:21.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:24:21.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:24:21.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-test 2024-09-18T16:24:21.138 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T16:24:21.138 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:24:21.138 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:24:21.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.562265 2024-09-18T16:24:21.141 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T16:24:21.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T16:24:21.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T16:24:21.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-09-18T16:24:21.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: TEST_interval_changes td/osd-scrub-test 2024-09-18T16:24:21.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:129: TEST_interval_changes: local poolname=test 2024-09-18T16:24:21.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:130: TEST_interval_changes: local OSDS=2 2024-09-18T16:24:21.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:131: TEST_interval_changes: local objects=10 2024-09-18T16:24:21.143 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:133: TEST_interval_changes: expr 24 '*' 60 '*' 60 2024-09-18T16:24:21.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:133: TEST_interval_changes: local day=86400 2024-09-18T16:24:21.145 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:134: TEST_interval_changes: expr 86400 '*' 7 2024-09-18T16:24:21.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:134: TEST_interval_changes: local week=604800 2024-09-18T16:24:21.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: TEST_interval_changes: local min_interval=86400 2024-09-18T16:24:21.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:136: TEST_interval_changes: local max_interval=604800 2024-09-18T16:24:21.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: TEST_interval_changes: local WAIT_FOR_UPDATE=15 2024-09-18T16:24:21.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: TEST_interval_changes: TESTDATA=testdata.562265 2024-09-18T16:24:21.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: TEST_interval_changes: run_mon td/osd-scrub-test a --osd_pool_default_size=2 2024-09-18T16:24:21.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-test 2024-09-18T16:24:21.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T16:24:21.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T16:24:21.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T16:24:21.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-test/a 2024-09-18T16:24:21.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=2 2024-09-18T16:24:21.520 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T16:24:21.520 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:24:21.520 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:24:21.521 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:24:21.521 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:24:21.521 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:24:21.521 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:24:21.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-18T16:24:21.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T16:24:21.794 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T16:24:21.794 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:24:21.794 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:24:21.794 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T16:24:21.794 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T16:24:21.795 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:24:21.796 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:24:21.796 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:24:21.796 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:24:21.796 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:24:21.797 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:24:21.797 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.562265/ceph-mon.a.asok 2024-09-18T16:24:21.798 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:24:21.798 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.562265/ceph-mon.a.asok config get fsid 2024-09-18T16:24:21.890 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T16:24:21.891 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:24:21.891 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:24:21.891 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T16:24:21.891 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T16:24:21.891 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:24:21.892 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:24:21.892 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:24:21.892 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:24:21.892 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:24:21.893 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:24:21.893 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.562265/ceph-mon.a.asok 2024-09-18T16:24:21.893 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:24:21.893 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.562265/ceph-mon.a.asok config get mon_host 2024-09-18T16:24:21.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:143: TEST_interval_changes: run_mgr td/osd-scrub-test x 2024-09-18T16:24:21.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-test 2024-09-18T16:24:21.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T16:24:21.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T16:24:21.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T16:24:21.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-test/x 2024-09-18T16:24:21.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T16:24:22.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T16:24:22.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:24:22.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:24:22.847 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:24:22.847 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:24:22.847 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:24:22.847 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:24:22.848 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:24:22.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:24:22.884 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: TEST_interval_changes: expr 2 - 1 2024-09-18T16:24:22.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: TEST_interval_changes: seq 0 1 2024-09-18T16:24:22.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: TEST_interval_changes: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:24:22.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:146: TEST_interval_changes: run_osd td/osd-scrub-test 0 --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2024-09-18T16:24:22.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-09-18T16:24:22.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:24:22.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T16:24:22.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:24:22.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/0 2024-09-18T16:24:22.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-18T16:24:22.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:24:22.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:24:22.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:24:22.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-09-18T16:24:22.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-09-18T16:24:22.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:24:22.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:24:22.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-18T16:24:22.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:24:22.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:24:22.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:24:22.891 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:24:22.891 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:24:22.891 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:24:22.891 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:24:22.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:24:22.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:24:22.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:24:22.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:24:22.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-18T16:24:22.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-18T16:24:22.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:24:22.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:24:22.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:24:22.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:24:22.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:24:22.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0' 2024-09-18T16:24:22.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/0 2024-09-18T16:24:22.893 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:24:22.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d286434c-d685-4317-8105-4ead12cc7699 2024-09-18T16:24:22.894 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 d286434c-d685-4317-8105-4ead12cc7699 2024-09-18T16:24:22.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d286434c-d685-4317-8105-4ead12cc7699' 2024-09-18T16:24:22.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:24:22.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC2/upmI4A6NhAAZSy2qLzTDglA9PX3esMYVQ== 2024-09-18T16:24:22.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC2/upmI4A6NhAAZSy2qLzTDglA9PX3esMYVQ=="}' 2024-09-18T16:24:22.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d286434c-d685-4317-8105-4ead12cc7699 -i td/osd-scrub-test/0/new.json 2024-09-18T16:24:23.194 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:24:23.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/0/new.json 2024-09-18T16:24:23.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 --mkfs --key AQC2/upmI4A6NhAAZSy2qLzTDglA9PX3esMYVQ== --osd-uuid d286434c-d685-4317-8105-4ead12cc7699 2024-09-18T16:24:23.238 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:24:23.235+0000 7f5dca820540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:24:23.241 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:24:23.235+0000 7f5dca820540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:24:23.243 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:24:23.239+0000 7f5dca820540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:24:23.244 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:24:23.239+0000 7f5dca820540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:24:23.244 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:24:23.239+0000 7f5dca820540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:24:23.244 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:24:23.239+0000 7f5dca820540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:24:23.326 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:24:23.323+0000 7f5dca820540 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-09-18T16:24:29.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-09-18T16:24:29.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:24:29.461 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T16:24:29.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T16:24:29.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:24:29.856 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T16:24:29.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T16:24:29.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:24:29.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:24:29.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2024-09-18T16:24:29.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:24:29.886 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:24:29.883+0000 7f7a60ad0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:24:29.889 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:24:29.883+0000 7f7a60ad0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:24:29.890 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:24:29.887+0000 7f7a60ad0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:24:30.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T16:24:30.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:24:30.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:24:30.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:24:30.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:24:30.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:24:30.145 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:24:30.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:24:30.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:24:30.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:24:30.444 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:24:30.439+0000 7f7a60ad0540 -1 Falling back to public interface 2024-09-18T16:24:30.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:24:31.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:24:31.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:24:31.462 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:24:31.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:24:31.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:24:31.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:24:31.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:24:32.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:24:32.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:24:32.776 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:24:32.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:24:32.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:24:32.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:24:33.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:24:33.087 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:24:33.083+0000 7f7a60ad0540 -1 osd.0 0 log_to_monitors true 2024-09-18T16:24:34.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:24:34.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:24:34.083 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:24:34.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:24:34.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:24:34.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:24:34.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:24:35.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:24:35.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:24:35.390 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:24:35.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:24:35.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:24:35.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:24:35.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:24:36.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:24:36.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:24:36.857 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:24:36.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:24:36.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:24:36.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:24:37.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:24:38.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:24:38.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:24:38.167 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:24:38.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:24:38.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:24:38.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:24:38.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:24:39.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:24:39.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:24:39.484 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:24:39.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:24:39.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:24:39.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:24:39.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:24:40.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:24:40.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:24:40.873 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:24:40.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:24:40.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:24:40.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:24:41.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:24:42.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:24:42.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:24:42.181 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:24:42.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:24:42.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:24:42.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:24:42.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:24:43.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:24:43.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:24:43.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:24:43.492 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:24:43.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:24:43.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:24:43.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:24:44.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:24:44.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:24:44.882 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:24:44.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:24:44.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:24:44.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:24:45.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:24:46.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:24:46.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:24:46.185 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:24:46.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:24:46.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:24:46.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:24:46.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:24:47.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:24:47.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:24:47.625 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:24:47.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:24:47.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:24:47.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:24:48.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:24:49.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:24:49.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:24:49.270 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:24:49.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:24:49.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:24:49.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:24:49.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:24:50.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:24:50.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:24:50.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:24:50.957 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:24:50.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:24:50.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:24:51.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:24:52.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:24:52.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:24:52.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:24:52.704 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:24:52.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:24:52.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:24:53.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:24:54.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:24:54.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:24:54.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:24:54.045 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:24:54.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:24:54.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:24:54.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:24:55.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:24:55.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:24:55.608 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:24:55.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:24:55.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:24:55.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:24:55.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:24:56.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:24:56.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:24:56.919 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T16:24:56.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T16:24:56.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:24:56.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:24:57.306 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/599787,v1:127.0.0.1:6803/599787] [v2:127.0.0.1:6804/599787,v1:127.0.0.1:6805/599787] exists,up d286434c-d685-4317-8105-4ead12cc7699 2024-09-18T16:24:57.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:24:57.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:24:57.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:24:57.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: TEST_interval_changes: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:24:57.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:146: TEST_interval_changes: run_osd td/osd-scrub-test 1 --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2024-09-18T16:24:57.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-09-18T16:24:57.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:24:57.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T16:24:57.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:24:57.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/1 2024-09-18T16:24:57.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-18T16:24:57.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:24:57.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:24:57.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:24:57.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-09-18T16:24:57.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-09-18T16:24:57.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:24:57.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:24:57.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-18T16:24:57.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:24:57.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:24:57.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:24:57.312 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:24:57.312 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:24:57.312 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:24:57.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:24:57.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:24:57.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:24:57.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:24:57.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:24:57.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-18T16:24:57.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-18T16:24:57.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:24:57.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:24:57.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:24:57.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:24:57.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:24:57.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0' 2024-09-18T16:24:57.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/1 2024-09-18T16:24:57.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:24:57.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=24ee5ef9-4caa-4765-8577-f77df8d036b0 2024-09-18T16:24:57.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 24ee5ef9-4caa-4765-8577-f77df8d036b0' 2024-09-18T16:24:57.314 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 24ee5ef9-4caa-4765-8577-f77df8d036b0 2024-09-18T16:24:57.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:24:57.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDZ/upmH9bVExAA2tiYWWTsMMfJ5dlsPFnyDQ== 2024-09-18T16:24:57.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDZ/upmH9bVExAA2tiYWWTsMMfJ5dlsPFnyDQ=="}' 2024-09-18T16:24:57.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 24ee5ef9-4caa-4765-8577-f77df8d036b0 -i td/osd-scrub-test/1/new.json 2024-09-18T16:24:57.739 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:24:57.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/1/new.json 2024-09-18T16:24:57.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 --mkfs --key AQDZ/upmH9bVExAA2tiYWWTsMMfJ5dlsPFnyDQ== --osd-uuid 24ee5ef9-4caa-4765-8577-f77df8d036b0 2024-09-18T16:24:57.789 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:24:57.787+0000 7ff27738d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:24:57.806 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:24:57.803+0000 7ff27738d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:24:57.808 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:24:57.803+0000 7ff27738d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:24:57.809 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:24:57.807+0000 7ff27738d540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:24:57.810 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:24:57.807+0000 7ff27738d540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:24:57.810 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:24:57.807+0000 7ff27738d540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:24:57.880 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:24:57.875+0000 7ff27738d540 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-09-18T16:25:03.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-09-18T16:25:03.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:25:03.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T16:25:03.865 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T16:25:03.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:25:04.274 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T16:25:04.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T16:25:04.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2024-09-18T16:25:04.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:25:04.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:25:04.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:25:04.307 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:25:04.303+0000 7f4205afb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:25:04.310 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:25:04.307+0000 7f4205afb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:25:04.312 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:25:04.307+0000 7f4205afb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:25:04.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T16:25:04.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:25:04.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T16:25:04.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:25:04.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:25:04.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:25:04.616 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:25:04.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:25:04.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:25:04.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:25:04.860 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:25:04.855+0000 7f4205afb540 -1 Falling back to public interface 2024-09-18T16:25:04.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:25:05.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:25:05.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:25:05.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:25:05.991 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:25:05.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:25:05.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:25:06.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:25:07.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:25:07.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:25:07.301 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:25:07.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:25:07.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:25:07.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:25:07.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:25:07.702 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:25:07.679+0000 7f4205afb540 -1 osd.1 0 log_to_monitors true 2024-09-18T16:25:08.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:25:08.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:25:08.608 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:25:08.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:25:08.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:25:08.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:25:09.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:25:10.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:25:10.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:25:10.058 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:25:10.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:25:10.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:25:10.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:25:10.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:25:11.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:25:11.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:25:11.421 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:25:11.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:25:11.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:25:11.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:25:11.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:25:12.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:25:12.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:25:12.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:25:12.733 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:25:12.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:25:12.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:25:13.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:25:14.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:25:14.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:25:14.046 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:25:14.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:25:14.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:25:14.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:25:14.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:25:15.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:25:15.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:25:15.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:25:15.359 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:25:15.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:25:15.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:25:15.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:25:16.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:25:16.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:25:16.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:25:16.766 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:25:16.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:25:16.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:25:17.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:25:18.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:25:18.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:25:18.078 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:25:18.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:25:18.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:25:18.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:25:18.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:25:19.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:25:19.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:25:19.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:25:19.445 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:25:19.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:25:19.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:25:19.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:25:20.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:25:20.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:25:20.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:25:20.756 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:25:20.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:25:20.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:25:21.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:25:22.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:25:22.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:25:22.270 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:25:22.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:25:22.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:25:22.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:25:22.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:25:23.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:25:23.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:25:23.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:25:23.707 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:25:23.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:25:23.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:25:24.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:25:25.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:25:25.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:25:25.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:25:25.088 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:25:25.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:25:25.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:25:25.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:25:26.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:25:26.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:25:26.470 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:25:26.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:25:26.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:25:26.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:25:27.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:25:28.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:25:28.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:25:28.060 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:25:28.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:25:28.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:25:28.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:25:28.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:25:29.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:25:29.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:25:29.419 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:25:29.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:25:29.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:25:29.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:25:30.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:25:31.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:25:31.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:25:31.062 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T16:25:31.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T16:25:31.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:25:31.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:25:31.373 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/601656,v1:127.0.0.1:6811/601656] [v2:127.0.0.1:6812/601656,v1:127.0.0.1:6813/601656] exists,up 24ee5ef9-4caa-4765-8577-f77df8d036b0 2024-09-18T16:25:31.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:25:31.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:25:31.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:25:31.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:150: TEST_interval_changes: create_pool test 1 1 2024-09-18T16:25:31.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-18T16:25:31.856 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'test' created 2024-09-18T16:25:31.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T16:25:32.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:151: TEST_interval_changes: wait_for_clean 2024-09-18T16:25:32.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:25:32.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:25:32.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:25:32.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:25:32.872 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:25:32.872 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:25:32.873 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:25:32.873 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:25:32.873 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:25:32.873 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:25:33.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:25:33.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:25:33.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:25:33.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=500 2024-09-18T16:25:33.092 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:25:33.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:25:33.451 INFO:tasks.workunit.client.0.smithi046.stderr:1' 2024-09-18T16:25:33.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:25:33.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:25:33.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:25:33.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836496 2024-09-18T16:25:33.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836496 2024-09-18T16:25:33.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836496' 2024-09-18T16:25:33.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:25:33.638 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:25:33.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672965 2024-09-18T16:25:33.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672965 2024-09-18T16:25:33.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836496 1-42949672965' 2024-09-18T16:25:33.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:25:33.774 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836496 2024-09-18T16:25:33.775 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:25:33.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:25:33.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836496 2024-09-18T16:25:33.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:25:33.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836496 2024-09-18T16:25:33.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-09-18T16:25:33.781 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836496 2024-09-18T16:25:33.781 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:25:34.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836495 -lt 21474836496 2024-09-18T16:25:34.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:25:35.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-18T16:25:35.139 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:25:35.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836496 -lt 21474836496 2024-09-18T16:25:35.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:25:35.452 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672965 2024-09-18T16:25:35.452 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:25:35.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:25:35.455 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672965 2024-09-18T16:25:35.455 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:25:35.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672965 2024-09-18T16:25:35.458 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672965 2024-09-18T16:25:35.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-18T16:25:35.458 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:25:35.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672965 -lt 42949672965 2024-09-18T16:25:35.786 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:25:35.786 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:25:35.787 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:25:36.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:25:36.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:25:36.201 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:25:36.201 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:25:36.201 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:25:36.201 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:25:36.202 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:25:36.202 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:25:36.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:25:36.490 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:25:36.491 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:25:36.491 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:25:36.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:25:36.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:25:36.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:25:36.839 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:152: TEST_interval_changes: ceph osd dump 2024-09-18T16:25:36.839 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:152: TEST_interval_changes: awk '{ print $2 }' 2024-09-18T16:25:36.839 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:152: TEST_interval_changes: grep '^pool.*['\'']test['\'']' 2024-09-18T16:25:37.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:152: TEST_interval_changes: local poolid=1 2024-09-18T16:25:37.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:154: TEST_interval_changes: dd if=/dev/urandom of=testdata.562265 bs=1032 count=1 2024-09-18T16:25:37.187 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records in 2024-09-18T16:25:37.188 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records out 2024-09-18T16:25:37.188 INFO:tasks.workunit.client.0.smithi046.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 0.000193025 s, 5.3 MB/s 2024-09-18T16:25:37.189 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:155: TEST_interval_changes: seq 1 10 2024-09-18T16:25:37.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:155: TEST_interval_changes: for i in `seq 1 $objects` 2024-09-18T16:25:37.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:157: TEST_interval_changes: rados -p test put obj1 testdata.562265 2024-09-18T16:25:37.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:155: TEST_interval_changes: for i in `seq 1 $objects` 2024-09-18T16:25:37.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:157: TEST_interval_changes: rados -p test put obj2 testdata.562265 2024-09-18T16:25:37.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:155: TEST_interval_changes: for i in `seq 1 $objects` 2024-09-18T16:25:37.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:157: TEST_interval_changes: rados -p test put obj3 testdata.562265 2024-09-18T16:25:37.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:155: TEST_interval_changes: for i in `seq 1 $objects` 2024-09-18T16:25:37.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:157: TEST_interval_changes: rados -p test put obj4 testdata.562265 2024-09-18T16:25:37.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:155: TEST_interval_changes: for i in `seq 1 $objects` 2024-09-18T16:25:37.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:157: TEST_interval_changes: rados -p test put obj5 testdata.562265 2024-09-18T16:25:37.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:155: TEST_interval_changes: for i in `seq 1 $objects` 2024-09-18T16:25:37.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:157: TEST_interval_changes: rados -p test put obj6 testdata.562265 2024-09-18T16:25:37.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:155: TEST_interval_changes: for i in `seq 1 $objects` 2024-09-18T16:25:37.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:157: TEST_interval_changes: rados -p test put obj7 testdata.562265 2024-09-18T16:25:38.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:155: TEST_interval_changes: for i in `seq 1 $objects` 2024-09-18T16:25:38.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:157: TEST_interval_changes: rados -p test put obj8 testdata.562265 2024-09-18T16:25:38.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:155: TEST_interval_changes: for i in `seq 1 $objects` 2024-09-18T16:25:38.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:157: TEST_interval_changes: rados -p test put obj9 testdata.562265 2024-09-18T16:25:38.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:155: TEST_interval_changes: for i in `seq 1 $objects` 2024-09-18T16:25:38.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:157: TEST_interval_changes: rados -p test put obj10 testdata.562265 2024-09-18T16:25:38.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:159: TEST_interval_changes: rm -f testdata.562265 2024-09-18T16:25:38.595 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:161: TEST_interval_changes: get_primary test obj1 2024-09-18T16:25:38.595 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T16:25:38.595 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T16:25:38.595 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T16:25:38.596 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T16:25:38.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:161: TEST_interval_changes: local primary=1 2024-09-18T16:25:38.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:164: TEST_interval_changes: check_dump_scrubs 1 '1 day' '1 week' 2024-09-18T16:25:38.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:115: check_dump_scrubs: local primary=1 2024-09-18T16:25:38.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:116: check_dump_scrubs: local 'sched_time_check=1 day' 2024-09-18T16:25:38.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:117: check_dump_scrubs: local 'deadline_check=1 week' 2024-09-18T16:25:38.906 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:119: check_dump_scrubs: get_asok_path osd.1 2024-09-18T16:25:38.906 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T16:25:38.906 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T16:25:38.907 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:25:38.907 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:25:38.907 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:25:38.907 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.562265/ceph-osd.1.asok 2024-09-18T16:25:38.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:119: check_dump_scrubs: CEPH_ARGS= 2024-09-18T16:25:38.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:119: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.562265/ceph-osd.1.asok dump_scrubs 2024-09-18T16:25:39.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:119: check_dump_scrubs: DS='[ 2024-09-18T16:25:39.000 INFO:tasks.workunit.client.0.smithi046.stderr: { 2024-09-18T16:25:39.000 INFO:tasks.workunit.client.0.smithi046.stderr: "pgid": "1.0", 2024-09-18T16:25:39.000 INFO:tasks.workunit.client.0.smithi046.stderr: "sched_time": "2024-09-19T16:25:31.761776+0000", 2024-09-18T16:25:39.000 INFO:tasks.workunit.client.0.smithi046.stderr: "deadline": "2024-09-25T16:25:31.761776+0000", 2024-09-18T16:25:39.001 INFO:tasks.workunit.client.0.smithi046.stderr: "forced": false 2024-09-18T16:25:39.001 INFO:tasks.workunit.client.0.smithi046.stderr: } 2024-09-18T16:25:39.001 INFO:tasks.workunit.client.0.smithi046.stderr:]' 2024-09-18T16:25:39.001 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-19T16:25:31.761776+0000",' '"deadline":' '"2024-09-25T16:25:31.761776+0000",' '"forced":' false '}' ']' 2024-09-18T16:25:39.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: check_dump_scrubs: jq '.[0].sched_time' 2024-09-18T16:25:39.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: check_dump_scrubs: eval 'SCHED_TIME="2024-09-19T16:25:31.761776+0000"' 2024-09-18T16:25:39.042 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: check_dump_scrubs: SCHED_TIME=2024-09-19T16:25:31.761776+0000 2024-09-18T16:25:39.043 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-09-18T16:25:39.043 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: check_dump_scrubs: echo 2024-09-19T16:25:31.761776+0000 2024-09-18T16:25:39.045 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: check_dump_scrubs: date +%Y-%m-%d -d 'now + 1 day' 2024-09-18T16:25:39.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: check_dump_scrubs: test 2024-09-19 = 2024-09-19 2024-09-18T16:25:39.048 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:124: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-19T16:25:31.761776+0000",' '"deadline":' '"2024-09-25T16:25:31.761776+0000",' '"forced":' false '}' ']' 2024-09-18T16:25:39.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:124: check_dump_scrubs: jq '.[0].deadline' 2024-09-18T16:25:39.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:124: check_dump_scrubs: eval 'DEADLINE="2024-09-25T16:25:31.761776+0000"' 2024-09-18T16:25:39.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:124: check_dump_scrubs: DEADLINE=2024-09-25T16:25:31.761776+0000 2024-09-18T16:25:39.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:125: check_dump_scrubs: echo 2024-09-25T16:25:31.761776+0000 2024-09-18T16:25:39.087 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:125: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-09-18T16:25:39.090 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:125: check_dump_scrubs: date +%Y-%m-%d -d 'now + 1 week' 2024-09-18T16:25:39.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:125: check_dump_scrubs: test 2024-09-25 = 2024-09-25 2024-09-18T16:25:39.093 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:167: TEST_interval_changes: get_asok_path osd.1 2024-09-18T16:25:39.093 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T16:25:39.093 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T16:25:39.094 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:25:39.094 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:25:39.094 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:25:39.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.562265/ceph-osd.1.asok 2024-09-18T16:25:39.095 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:167: TEST_interval_changes: expr 86400 '*' 2 2024-09-18T16:25:39.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:167: TEST_interval_changes: CEPH_ARGS= 2024-09-18T16:25:39.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:167: TEST_interval_changes: ceph --admin-daemon /tmp/ceph-asok.562265/ceph-osd.1.asok config set osd_scrub_min_interval 172800 2024-09-18T16:25:39.183 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:25:39.183 INFO:tasks.workunit.client.0.smithi046.stdout: "success": "osd_scrub_min_interval = '' " 2024-09-18T16:25:39.183 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:25:39.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: sleep 15 2024-09-18T16:25:54.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:169: TEST_interval_changes: check_dump_scrubs 1 '2 days' '1 week' 2024-09-18T16:25:54.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:115: check_dump_scrubs: local primary=1 2024-09-18T16:25:54.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:116: check_dump_scrubs: local 'sched_time_check=2 days' 2024-09-18T16:25:54.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:117: check_dump_scrubs: local 'deadline_check=1 week' 2024-09-18T16:25:54.217 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:119: check_dump_scrubs: get_asok_path osd.1 2024-09-18T16:25:54.217 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T16:25:54.217 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T16:25:54.217 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:25:54.217 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:25:54.217 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:25:54.217 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.562265/ceph-osd.1.asok 2024-09-18T16:25:54.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:119: check_dump_scrubs: CEPH_ARGS= 2024-09-18T16:25:54.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:119: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.562265/ceph-osd.1.asok dump_scrubs 2024-09-18T16:25:54.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:119: check_dump_scrubs: DS='[ 2024-09-18T16:25:54.297 INFO:tasks.workunit.client.0.smithi046.stderr: { 2024-09-18T16:25:54.297 INFO:tasks.workunit.client.0.smithi046.stderr: "pgid": "1.0", 2024-09-18T16:25:54.297 INFO:tasks.workunit.client.0.smithi046.stderr: "sched_time": "2024-09-20T16:25:31.761776+0000", 2024-09-18T16:25:54.298 INFO:tasks.workunit.client.0.smithi046.stderr: "deadline": "2024-09-25T16:25:31.761776+0000", 2024-09-18T16:25:54.298 INFO:tasks.workunit.client.0.smithi046.stderr: "forced": false 2024-09-18T16:25:54.298 INFO:tasks.workunit.client.0.smithi046.stderr: } 2024-09-18T16:25:54.298 INFO:tasks.workunit.client.0.smithi046.stderr:]' 2024-09-18T16:25:54.299 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-20T16:25:31.761776+0000",' '"deadline":' '"2024-09-25T16:25:31.761776+0000",' '"forced":' false '}' ']' 2024-09-18T16:25:54.299 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: check_dump_scrubs: jq '.[0].sched_time' 2024-09-18T16:25:54.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: check_dump_scrubs: eval 'SCHED_TIME="2024-09-20T16:25:31.761776+0000"' 2024-09-18T16:25:54.341 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: check_dump_scrubs: SCHED_TIME=2024-09-20T16:25:31.761776+0000 2024-09-18T16:25:54.342 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: check_dump_scrubs: echo 2024-09-20T16:25:31.761776+0000 2024-09-18T16:25:54.343 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-09-18T16:25:54.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: check_dump_scrubs: date +%Y-%m-%d -d 'now + 2 days' 2024-09-18T16:25:54.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: check_dump_scrubs: test 2024-09-20 = 2024-09-20 2024-09-18T16:25:54.348 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:124: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-20T16:25:31.761776+0000",' '"deadline":' '"2024-09-25T16:25:31.761776+0000",' '"forced":' false '}' ']' 2024-09-18T16:25:54.348 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:124: check_dump_scrubs: jq '.[0].deadline' 2024-09-18T16:25:54.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:124: check_dump_scrubs: eval 'DEADLINE="2024-09-25T16:25:31.761776+0000"' 2024-09-18T16:25:54.392 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:124: check_dump_scrubs: DEADLINE=2024-09-25T16:25:31.761776+0000 2024-09-18T16:25:54.393 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:125: check_dump_scrubs: echo 2024-09-25T16:25:31.761776+0000 2024-09-18T16:25:54.393 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:125: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-09-18T16:25:54.396 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:125: check_dump_scrubs: date +%Y-%m-%d -d 'now + 1 week' 2024-09-18T16:25:54.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:125: check_dump_scrubs: test 2024-09-25 = 2024-09-25 2024-09-18T16:25:54.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:172: TEST_interval_changes: get_asok_path osd.1 2024-09-18T16:25:54.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T16:25:54.399 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T16:25:54.400 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:25:54.400 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:25:54.400 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:25:54.400 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.562265/ceph-osd.1.asok 2024-09-18T16:25:54.401 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:172: TEST_interval_changes: expr 604800 '*' 2 2024-09-18T16:25:54.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:172: TEST_interval_changes: CEPH_ARGS= 2024-09-18T16:25:54.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:172: TEST_interval_changes: ceph --admin-daemon /tmp/ceph-asok.562265/ceph-osd.1.asok config set osd_scrub_max_interval 1209600 2024-09-18T16:25:54.488 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:25:54.488 INFO:tasks.workunit.client.0.smithi046.stdout: "success": "osd_scrub_max_interval = '' " 2024-09-18T16:25:54.488 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:25:54.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:173: TEST_interval_changes: sleep 15 2024-09-18T16:26:09.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:174: TEST_interval_changes: check_dump_scrubs 1 '2 days' '2 week' 2024-09-18T16:26:09.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:115: check_dump_scrubs: local primary=1 2024-09-18T16:26:09.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:116: check_dump_scrubs: local 'sched_time_check=2 days' 2024-09-18T16:26:09.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:117: check_dump_scrubs: local 'deadline_check=2 week' 2024-09-18T16:26:09.499 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:119: check_dump_scrubs: get_asok_path osd.1 2024-09-18T16:26:09.499 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T16:26:09.499 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T16:26:09.500 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:26:09.500 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:26:09.500 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:26:09.501 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.562265/ceph-osd.1.asok 2024-09-18T16:26:09.501 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:119: check_dump_scrubs: CEPH_ARGS= 2024-09-18T16:26:09.501 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:119: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.562265/ceph-osd.1.asok dump_scrubs 2024-09-18T16:26:09.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:119: check_dump_scrubs: DS='[ 2024-09-18T16:26:09.602 INFO:tasks.workunit.client.0.smithi046.stderr: { 2024-09-18T16:26:09.602 INFO:tasks.workunit.client.0.smithi046.stderr: "pgid": "1.0", 2024-09-18T16:26:09.603 INFO:tasks.workunit.client.0.smithi046.stderr: "sched_time": "2024-09-20T16:25:31.761776+0000", 2024-09-18T16:26:09.603 INFO:tasks.workunit.client.0.smithi046.stderr: "deadline": "2024-10-02T16:25:31.761776+0000", 2024-09-18T16:26:09.603 INFO:tasks.workunit.client.0.smithi046.stderr: "forced": false 2024-09-18T16:26:09.603 INFO:tasks.workunit.client.0.smithi046.stderr: } 2024-09-18T16:26:09.603 INFO:tasks.workunit.client.0.smithi046.stderr:]' 2024-09-18T16:26:09.604 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-20T16:25:31.761776+0000",' '"deadline":' '"2024-10-02T16:25:31.761776+0000",' '"forced":' false '}' ']' 2024-09-18T16:26:09.604 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: check_dump_scrubs: jq '.[0].sched_time' 2024-09-18T16:26:09.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: check_dump_scrubs: eval 'SCHED_TIME="2024-09-20T16:25:31.761776+0000"' 2024-09-18T16:26:09.648 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: check_dump_scrubs: SCHED_TIME=2024-09-20T16:25:31.761776+0000 2024-09-18T16:26:09.649 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: check_dump_scrubs: echo 2024-09-20T16:25:31.761776+0000 2024-09-18T16:26:09.649 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-09-18T16:26:09.653 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: check_dump_scrubs: date +%Y-%m-%d -d 'now + 2 days' 2024-09-18T16:26:09.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: check_dump_scrubs: test 2024-09-20 = 2024-09-20 2024-09-18T16:26:09.654 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:124: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-20T16:25:31.761776+0000",' '"deadline":' '"2024-10-02T16:25:31.761776+0000",' '"forced":' false '}' ']' 2024-09-18T16:26:09.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:124: check_dump_scrubs: jq '.[0].deadline' 2024-09-18T16:26:09.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:124: check_dump_scrubs: eval 'DEADLINE="2024-10-02T16:25:31.761776+0000"' 2024-09-18T16:26:09.696 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:124: check_dump_scrubs: DEADLINE=2024-10-02T16:25:31.761776+0000 2024-09-18T16:26:09.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:125: check_dump_scrubs: echo 2024-10-02T16:25:31.761776+0000 2024-09-18T16:26:09.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:125: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-09-18T16:26:09.702 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:125: check_dump_scrubs: date +%Y-%m-%d -d 'now + 2 week' 2024-09-18T16:26:09.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:125: check_dump_scrubs: test 2024-10-02 = 2024-10-02 2024-09-18T16:26:09.704 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: expr 86400 '*' 3 2024-09-18T16:26:09.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: ceph osd pool set test scrub_min_interval 259200 2024-09-18T16:26:10.137 INFO:tasks.workunit.client.0.smithi046.stderr:set pool 1 scrub_min_interval to 259200 2024-09-18T16:26:10.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:178: TEST_interval_changes: sleep 15 2024-09-18T16:26:25.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:179: TEST_interval_changes: check_dump_scrubs 1 '3 days' '2 week' 2024-09-18T16:26:25.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:115: check_dump_scrubs: local primary=1 2024-09-18T16:26:25.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:116: check_dump_scrubs: local 'sched_time_check=3 days' 2024-09-18T16:26:25.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:117: check_dump_scrubs: local 'deadline_check=2 week' 2024-09-18T16:26:25.331 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:119: check_dump_scrubs: get_asok_path osd.1 2024-09-18T16:26:25.331 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T16:26:25.331 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T16:26:25.332 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:26:25.332 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:26:25.332 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:26:25.332 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.562265/ceph-osd.1.asok 2024-09-18T16:26:25.332 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:119: check_dump_scrubs: CEPH_ARGS= 2024-09-18T16:26:25.332 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:119: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.562265/ceph-osd.1.asok dump_scrubs 2024-09-18T16:26:25.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:119: check_dump_scrubs: DS='[ 2024-09-18T16:26:25.332 INFO:tasks.workunit.client.0.smithi046.stderr: { 2024-09-18T16:26:25.332 INFO:tasks.workunit.client.0.smithi046.stderr: "pgid": "1.0", 2024-09-18T16:26:25.332 INFO:tasks.workunit.client.0.smithi046.stderr: "sched_time": "2024-09-21T16:25:31.761776+0000", 2024-09-18T16:26:25.333 INFO:tasks.workunit.client.0.smithi046.stderr: "deadline": "2024-10-02T16:25:31.761776+0000", 2024-09-18T16:26:25.333 INFO:tasks.workunit.client.0.smithi046.stderr: "forced": false 2024-09-18T16:26:25.333 INFO:tasks.workunit.client.0.smithi046.stderr: } 2024-09-18T16:26:25.333 INFO:tasks.workunit.client.0.smithi046.stderr:]' 2024-09-18T16:26:25.333 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-21T16:25:31.761776+0000",' '"deadline":' '"2024-10-02T16:25:31.761776+0000",' '"forced":' false '}' ']' 2024-09-18T16:26:25.333 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: check_dump_scrubs: jq '.[0].sched_time' 2024-09-18T16:26:25.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: check_dump_scrubs: eval 'SCHED_TIME="2024-09-21T16:25:31.761776+0000"' 2024-09-18T16:26:25.333 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: check_dump_scrubs: SCHED_TIME=2024-09-21T16:25:31.761776+0000 2024-09-18T16:26:25.333 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: check_dump_scrubs: echo 2024-09-21T16:25:31.761776+0000 2024-09-18T16:26:25.333 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-09-18T16:26:25.334 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: check_dump_scrubs: date +%Y-%m-%d -d 'now + 3 days' 2024-09-18T16:26:25.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: check_dump_scrubs: test 2024-09-21 = 2024-09-21 2024-09-18T16:26:25.334 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:124: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-21T16:25:31.761776+0000",' '"deadline":' '"2024-10-02T16:25:31.761776+0000",' '"forced":' false '}' ']' 2024-09-18T16:26:25.334 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:124: check_dump_scrubs: jq '.[0].deadline' 2024-09-18T16:26:25.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:124: check_dump_scrubs: eval 'DEADLINE="2024-10-02T16:25:31.761776+0000"' 2024-09-18T16:26:25.356 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:124: check_dump_scrubs: DEADLINE=2024-10-02T16:25:31.761776+0000 2024-09-18T16:26:25.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:125: check_dump_scrubs: echo 2024-10-02T16:25:31.761776+0000 2024-09-18T16:26:25.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:125: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-09-18T16:26:25.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:125: check_dump_scrubs: date +%Y-%m-%d -d 'now + 2 week' 2024-09-18T16:26:25.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:125: check_dump_scrubs: test 2024-10-02 = 2024-10-02 2024-09-18T16:26:25.362 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:182: TEST_interval_changes: expr 604800 '*' 3 2024-09-18T16:26:25.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:182: TEST_interval_changes: ceph osd pool set test scrub_max_interval 1814400 2024-09-18T16:26:25.792 INFO:tasks.workunit.client.0.smithi046.stderr:set pool 1 scrub_max_interval to 1814400 2024-09-18T16:26:25.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:183: TEST_interval_changes: sleep 15 2024-09-18T16:26:40.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:184: TEST_interval_changes: check_dump_scrubs 1 '3 days' '3 week' 2024-09-18T16:26:40.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:115: check_dump_scrubs: local primary=1 2024-09-18T16:26:40.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:116: check_dump_scrubs: local 'sched_time_check=3 days' 2024-09-18T16:26:40.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:117: check_dump_scrubs: local 'deadline_check=3 week' 2024-09-18T16:26:40.808 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:119: check_dump_scrubs: get_asok_path osd.1 2024-09-18T16:26:40.808 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T16:26:40.808 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T16:26:40.809 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:26:40.809 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:26:40.809 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:26:40.809 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.562265/ceph-osd.1.asok 2024-09-18T16:26:40.810 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:119: check_dump_scrubs: CEPH_ARGS= 2024-09-18T16:26:40.810 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:119: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.562265/ceph-osd.1.asok dump_scrubs 2024-09-18T16:26:40.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:119: check_dump_scrubs: DS='[ 2024-09-18T16:26:40.909 INFO:tasks.workunit.client.0.smithi046.stderr: { 2024-09-18T16:26:40.909 INFO:tasks.workunit.client.0.smithi046.stderr: "pgid": "1.0", 2024-09-18T16:26:40.910 INFO:tasks.workunit.client.0.smithi046.stderr: "sched_time": "2024-09-21T16:25:31.761776+0000", 2024-09-18T16:26:40.910 INFO:tasks.workunit.client.0.smithi046.stderr: "deadline": "2024-10-09T16:25:31.761776+0000", 2024-09-18T16:26:40.910 INFO:tasks.workunit.client.0.smithi046.stderr: "forced": false 2024-09-18T16:26:40.910 INFO:tasks.workunit.client.0.smithi046.stderr: } 2024-09-18T16:26:40.910 INFO:tasks.workunit.client.0.smithi046.stderr:]' 2024-09-18T16:26:40.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-21T16:25:31.761776+0000",' '"deadline":' '"2024-10-09T16:25:31.761776+0000",' '"forced":' false '}' ']' 2024-09-18T16:26:40.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: check_dump_scrubs: jq '.[0].sched_time' 2024-09-18T16:26:40.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: check_dump_scrubs: eval 'SCHED_TIME="2024-09-21T16:25:31.761776+0000"' 2024-09-18T16:26:40.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: check_dump_scrubs: SCHED_TIME=2024-09-21T16:25:31.761776+0000 2024-09-18T16:26:40.956 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: check_dump_scrubs: echo 2024-09-21T16:25:31.761776+0000 2024-09-18T16:26:40.956 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-09-18T16:26:40.959 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: check_dump_scrubs: date +%Y-%m-%d -d 'now + 3 days' 2024-09-18T16:26:40.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: check_dump_scrubs: test 2024-09-21 = 2024-09-21 2024-09-18T16:26:40.962 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:124: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-21T16:25:31.761776+0000",' '"deadline":' '"2024-10-09T16:25:31.761776+0000",' '"forced":' false '}' ']' 2024-09-18T16:26:40.962 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:124: check_dump_scrubs: jq '.[0].deadline' 2024-09-18T16:26:41.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:124: check_dump_scrubs: eval 'DEADLINE="2024-10-09T16:25:31.761776+0000"' 2024-09-18T16:26:41.004 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:124: check_dump_scrubs: DEADLINE=2024-10-09T16:25:31.761776+0000 2024-09-18T16:26:41.005 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:125: check_dump_scrubs: echo 2024-10-09T16:25:31.761776+0000 2024-09-18T16:26:41.005 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:125: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-09-18T16:26:41.009 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:125: check_dump_scrubs: date +%Y-%m-%d -d 'now + 3 week' 2024-09-18T16:26:41.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:125: check_dump_scrubs: test 2024-10-09 = 2024-10-09 2024-09-18T16:26:41.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: teardown td/osd-scrub-test 2024-09-18T16:26:41.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-09-18T16:26:41.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:26:41.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-18T16:26:41.011 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:26:41.011 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:26:41.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:26:41.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:26:41.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:26:41.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:26:41.127 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:26:41.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:26:41.129 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:26:41.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:26:41.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:26:41.132 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:26:41.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:26:41.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:26:41.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:26:41.135 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:26:41.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:26:41.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:26:41.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-09-18T16:26:41.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:26:41.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:26:41.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:26:41.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.562265 2024-09-18T16:26:41.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:26:41.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:26:41.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-09-18T16:26:41.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: setup td/osd-scrub-test 2024-09-18T16:26:41.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-test 2024-09-18T16:26:41.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-test 2024-09-18T16:26:41.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-09-18T16:26:41.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:26:41.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-18T16:26:41.159 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:26:41.159 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:26:41.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:26:41.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:26:41.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:26:41.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:26:41.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:26:41.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:26:41.165 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:26:41.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:26:41.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:26:41.167 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:26:41.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:26:41.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:26:41.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:26:41.170 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:26:41.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:26:41.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:26:41.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-09-18T16:26:41.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:26:41.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:26:41.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:26:41.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.562265 2024-09-18T16:26:41.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:26:41.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:26:41.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-test 2024-09-18T16:26:41.180 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T16:26:41.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:26:41.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:26:41.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.562265 2024-09-18T16:26:41.183 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T16:26:41.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T16:26:41.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T16:26:41.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-09-18T16:26:41.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: TEST_just_deep_scrubs td/osd-scrub-test 2024-09-18T16:26:41.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:448: TEST_just_deep_scrubs: local dir=td/osd-scrub-test 2024-09-18T16:26:41.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_just_deep_scrubs: cluster_conf=(['osds_num']="3" ['pgs_in_pool']="4" ['pool_name']="test") 2024-09-18T16:26:41.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_just_deep_scrubs: local -A cluster_conf 2024-09-18T16:26:41.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:455: TEST_just_deep_scrubs: standard_scrub_cluster td/osd-scrub-test cluster_conf 2024-09-18T16:26:41.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-scrub-test 2024-09-18T16:26:41.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-09-18T16:26:41.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-09-18T16:26:41.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=4 2024-09-18T16:26:41.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=test 2024-09-18T16:26:41.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=test 2024-09-18T16:26:41.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local extra_pars= 2024-09-18T16:26:41.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-09-18T16:26:41.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-09-18T16:26:41.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-09-18T16:26:42.926 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 9bdf5d34-9122-45bc-b753-12be2e42fa07 2024-09-18T16:26:43.238 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:26:43.283 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:26:43.278+0000 7f4589d80540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:26:43.287 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:26:43.282+0000 7f4589d80540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:26:43.289 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:26:43.286+0000 7f4589d80540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:26:43.290 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:26:43.286+0000 7f4589d80540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:26:43.290 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:26:43.286+0000 7f4589d80540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:26:43.290 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:26:43.286+0000 7f4589d80540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:26:43.371 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:26:43.366+0000 7f4589d80540 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-09-18T16:26:48.987 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T16:26:49.369 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T16:26:49.401 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:26:49.393+0000 7f18a397a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:26:49.403 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:26:49.397+0000 7f18a397a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:26:49.405 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:26:49.397+0000 7f18a397a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:26:49.720 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:26:49.951 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:26:49.945+0000 7f18a397a540 -1 Falling back to public interface 2024-09-18T16:26:51.065 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:26:52.372 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:26:52.636 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:26:52.629+0000 7f18a397a540 -1 osd.0 0 log_to_monitors true 2024-09-18T16:26:53.691 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:26:55.004 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:26:56.371 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:26:57.683 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:26:58.999 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:27:00.354 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:27:01.674 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:27:02.989 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:27:04.340 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:27:05.733 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:27:07.230 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:27:08.922 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:27:10.599 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:27:12.162 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:27:13.784 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:27:15.192 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:27:15.493 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/604524,v1:127.0.0.1:6803/604524] [v2:127.0.0.1:6804/604524,v1:127.0.0.1:6805/604524] exists,up 9bdf5d34-9122-45bc-b753-12be2e42fa07 2024-09-18T16:27:15.501 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 ab5762de-c44c-481f-ab17-1e9a3dc6360d 2024-09-18T16:27:15.913 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:27:15.958 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:27:15.953+0000 7fada8ec3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:27:15.970 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:27:15.965+0000 7fada8ec3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:27:15.973 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:27:15.965+0000 7fada8ec3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:27:15.973 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:27:15.969+0000 7fada8ec3540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:27:15.973 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:27:15.969+0000 7fada8ec3540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:27:15.973 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:27:15.969+0000 7fada8ec3540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:27:16.046 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:27:16.041+0000 7fada8ec3540 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-09-18T16:27:22.324 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T16:27:22.712 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T16:27:22.744 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:27:22.737+0000 7fb8f964c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:27:22.747 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:27:22.741+0000 7fb8f964c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:27:22.754 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:27:22.741+0000 7fb8f964c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:27:23.085 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:27:23.300 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:27:23.293+0000 7fb8f964c540 -1 Falling back to public interface 2024-09-18T16:27:24.433 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:27:25.749 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:27:26.041 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:27:26.033+0000 7fb8f964c540 -1 osd.1 0 log_to_monitors true 2024-09-18T16:27:27.055 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:27:28.493 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:27:29.827 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:27:31.153 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:27:32.510 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:27:33.895 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:27:35.205 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:27:36.520 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:27:37.839 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:27:39.149 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:27:40.678 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:27:42.313 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:27:43.960 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:27:45.596 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:27:47.138 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:27:48.450 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:27:49.758 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T16:27:50.075 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/606304,v1:127.0.0.1:6811/606304] [v2:127.0.0.1:6812/606304,v1:127.0.0.1:6813/606304] exists,up ab5762de-c44c-481f-ab17-1e9a3dc6360d 2024-09-18T16:27:50.081 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 081c54d7-642e-4cf8-ac1b-cb949ff2ffe6 2024-09-18T16:27:50.502 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:27:50.554 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:27:50.549+0000 7f2ae2aa5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:27:50.567 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:27:50.561+0000 7f2ae2aa5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:27:50.569 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:27:50.565+0000 7f2ae2aa5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:27:50.570 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:27:50.565+0000 7f2ae2aa5540 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-09-18T16:27:50.570 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:27:50.565+0000 7f2ae2aa5540 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-09-18T16:27:50.571 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:27:50.565+0000 7f2ae2aa5540 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-09-18T16:27:50.642 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:27:50.637+0000 7f2ae2aa5540 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-09-18T16:27:56.868 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T16:27:57.279 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T16:27:57.311 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:27:57.305+0000 7ff6c5136540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:27:57.314 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:27:57.309+0000 7ff6c5136540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:27:57.318 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:27:57.309+0000 7ff6c5136540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:27:57.565 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:27:57.856 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:27:57.849+0000 7ff6c5136540 -1 Falling back to public interface 2024-09-18T16:27:58.916 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:28:00.223 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:28:00.714 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:28:00.709+0000 7ff6c5136540 -1 osd.2 0 log_to_monitors true 2024-09-18T16:28:01.537 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:28:02.982 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:28:04.431 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:28:05.861 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:28:07.171 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:28:08.593 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:28:09.900 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:28:11.211 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:28:12.630 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:28:13.945 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:28:15.542 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:28:17.232 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:28:18.649 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:28:20.145 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:28:21.593 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:28:22.952 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:28:24.278 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T16:28:24.656 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/608109,v1:127.0.0.1:6819/608109] [v2:127.0.0.1:6820/608109,v1:127.0.0.1:6821/608109] exists,up 081c54d7-642e-4cf8-ac1b-cb949ff2ffe6 2024-09-18T16:28:25.105 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'test' created 2024-09-18T16:28:27.522 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836504 2024-09-18T16:28:29.173 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672973 2024-09-18T16:28:29.526 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509448 2024-09-18T16:28:32.248 INFO:tasks.workunit.client.0.smithi046.stdout:standard_scrub_cluster: dbg: test pool is test 1 2024-09-18T16:28:32.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_just_deep_scrubs: local poolid=1 2024-09-18T16:28:32.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_just_deep_scrubs: local poolname=test 2024-09-18T16:28:32.249 INFO:tasks.workunit.client.0.smithi046.stdout:Pool: test : 1 2024-09-18T16:28:32.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:458: TEST_just_deep_scrubs: echo 'Pool: test : 1' 2024-09-18T16:28:32.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_just_deep_scrubs: TESTDATA=testdata.562265 2024-09-18T16:28:32.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:461: TEST_just_deep_scrubs: local objects=15 2024-09-18T16:28:32.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:462: TEST_just_deep_scrubs: dd if=/dev/urandom of=testdata.562265 bs=1032 count=1 2024-09-18T16:28:32.250 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records in 2024-09-18T16:28:32.250 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records out 2024-09-18T16:28:32.250 INFO:tasks.workunit.client.0.smithi046.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 0.000193938 s, 5.3 MB/s 2024-09-18T16:28:32.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:463: TEST_just_deep_scrubs: seq 1 15 2024-09-18T16:28:32.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:463: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-18T16:28:32.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:465: TEST_just_deep_scrubs: rados -p test put obj1 testdata.562265 2024-09-18T16:28:32.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:463: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-18T16:28:32.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:465: TEST_just_deep_scrubs: rados -p test put obj2 testdata.562265 2024-09-18T16:28:32.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:463: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-18T16:28:32.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:465: TEST_just_deep_scrubs: rados -p test put obj3 testdata.562265 2024-09-18T16:28:32.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:463: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-18T16:28:32.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:465: TEST_just_deep_scrubs: rados -p test put obj4 testdata.562265 2024-09-18T16:28:32.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:463: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-18T16:28:32.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:465: TEST_just_deep_scrubs: rados -p test put obj5 testdata.562265 2024-09-18T16:28:33.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:463: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-18T16:28:33.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:465: TEST_just_deep_scrubs: rados -p test put obj6 testdata.562265 2024-09-18T16:28:33.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:463: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-18T16:28:33.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:465: TEST_just_deep_scrubs: rados -p test put obj7 testdata.562265 2024-09-18T16:28:33.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:463: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-18T16:28:33.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:465: TEST_just_deep_scrubs: rados -p test put obj8 testdata.562265 2024-09-18T16:28:33.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:463: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-18T16:28:33.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:465: TEST_just_deep_scrubs: rados -p test put obj9 testdata.562265 2024-09-18T16:28:33.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:463: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-18T16:28:33.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:465: TEST_just_deep_scrubs: rados -p test put obj10 testdata.562265 2024-09-18T16:28:33.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:463: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-18T16:28:33.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:465: TEST_just_deep_scrubs: rados -p test put obj11 testdata.562265 2024-09-18T16:28:34.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:463: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-18T16:28:34.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:465: TEST_just_deep_scrubs: rados -p test put obj12 testdata.562265 2024-09-18T16:28:34.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:463: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-18T16:28:34.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:465: TEST_just_deep_scrubs: rados -p test put obj13 testdata.562265 2024-09-18T16:28:34.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:463: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-18T16:28:34.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:465: TEST_just_deep_scrubs: rados -p test put obj14 testdata.562265 2024-09-18T16:28:34.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:463: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-18T16:28:34.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:465: TEST_just_deep_scrubs: rados -p test put obj15 testdata.562265 2024-09-18T16:28:34.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:467: TEST_just_deep_scrubs: rm -f testdata.562265 2024-09-18T16:28:34.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:472: TEST_just_deep_scrubs: ceph osd set noscrub 2024-09-18T16:28:35.154 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T16:28:35.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:473: TEST_just_deep_scrubs: sleep 6 2024-09-18T16:28:41.169 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:474: TEST_just_deep_scrubs: date -Ins 2024-09-18T16:28:41.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:474: TEST_just_deep_scrubs: local now_is=2024-09-18T16:28:41,169715147+00:00 2024-09-18T16:28:41.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:475: TEST_just_deep_scrubs: declare -A sched_data 2024-09-18T16:28:41.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:476: TEST_just_deep_scrubs: local pgid=1.2 2024-09-18T16:28:41.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: set_query_debug 1.2 2024-09-18T16:28:41.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:293: set_query_debug: local pgid=1.2 2024-09-18T16:28:41.172 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:295: set_query_debug: ceph pg dump pgs_brief 2024-09-18T16:28:41.173 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:295: set_query_debug: awk -v 'pg=^1.2' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2024-09-18T16:28:41.464 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs_brief 2024-09-18T16:28:41.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:295: set_query_debug: local prim_osd=0 2024-09-18T16:28:41.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: echo 'Setting scrub debug data. Primary for 1.2 is 0' 2024-09-18T16:28:41.474 INFO:tasks.workunit.client.0.smithi046.stdout:Setting scrub debug data. Primary for 1.2 is 0 2024-09-18T16:28:41.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:298: set_query_debug: get_asok_path osd.0 2024-09-18T16:28:41.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T16:28:41.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T16:28:41.475 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:28:41.475 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:28:41.476 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:28:41.476 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.562265/ceph-osd.0.asok 2024-09-18T16:28:41.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:298: set_query_debug: CEPH_ARGS= 2024-09-18T16:28:41.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:298: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.562265/ceph-osd.0.asok scrubdebug 1.2 set sessions 2024-09-18T16:28:41.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:481: TEST_just_deep_scrubs: extract_published_sch 1.2 2024-09-18T16:28:41,169715147+00:00 2024-09-18T16:28:41,169715147+00:00 sched_data 2024-09-18T16:28:41.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.2 2024-09-18T16:28:41.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sched_data 2024-09-18T16:28:41.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2024-09-18T16:28:41,169715147+00:00 2024-09-18T16:28:41.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2024-09-18T16:28:41,169715147+00:00 2024-09-18T16:28:41.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=1 2024-09-18T16:28:41.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2024-09-18T16:28:41.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2024-09-18T16:28:41.856 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:28:42.043 INFO:tasks.workunit.client.0.smithi046.stdout:{"success":true}( 2024-09-18T16:28:42.043 INFO:tasks.workunit.client.0.smithi046.stdout:[query_epoch]=23 2024-09-18T16:28:42.043 INFO:tasks.workunit.client.0.smithi046.stdout:[query_seq]=14 2024-09-18T16:28:42.043 INFO:tasks.workunit.client.0.smithi046.stdout:[query_active]=false 2024-09-18T16:28:42.043 INFO:tasks.workunit.client.0.smithi046.stdout:[query_schedule]='scrub scheduled' 2024-09-18T16:28:42.043 INFO:tasks.workunit.client.0.smithi046.stdout:[query_schedule_at]='2024-09-19T16:28:25.022552+0000' 2024-09-18T16:28:42.043 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_duration]=0 2024-09-18T16:28:42.043 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_stamp]='2024-09-18T16:28:25.022552+0000' 2024-09-18T16:28:42.043 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_scrub]='0x0' 2024-09-18T16:28:42.043 INFO:tasks.workunit.client.0.smithi046.stdout:[query_is_future]=true 2024-09-18T16:28:42.043 INFO:tasks.workunit.client.0.smithi046.stdout:[query_vs_date]=true 2024-09-18T16:28:42.043 INFO:tasks.workunit.client.0.smithi046.stdout:[query_scrub_seq]=909312 2024-09-18T16:28:42.044 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_pg_state]='active+clean' 2024-09-18T16:28:42.044 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_state_has_scrubbing]=false 2024-09-18T16:28:42.044 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_last_duration]=0 2024-09-18T16:28:42.044 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-18T16:28:42.044 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_schedule_at]='2024-09-19T16:28:25.022552+0000' 2024-09-18T16:28:42.044 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_is_future]=true 2024-09-18T16:28:42.044 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_vs_date]=true 2024-09-18T16:28:42.044 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_reported_epoch]=22 2024-09-18T16:28:42.044 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_seq]=13 2024-09-18T16:28:42.044 INFO:tasks.workunit.client.0.smithi046.stdout:) 2024-09-18T16:28:42.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: local saved_last_stamp=2024-09-18T16:28:25.022552+0000 2024-09-18T16:28:42.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:483: TEST_just_deep_scrubs: local dbg_counter_at_start=909312 2024-09-18T16:28:42.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: echo 'test counter @ start: 909312' 2024-09-18T16:28:42.089 INFO:tasks.workunit.client.0.smithi046.stdout:test counter @ start: 909312 2024-09-18T16:28:42.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:486: TEST_just_deep_scrubs: ceph pg 1.2 deep_scrub 2024-09-18T16:28:42.206 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:28:42.207 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": true, 2024-09-18T16:28:42.207 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T16:27:02.205419+0000" 2024-09-18T16:28:42.207 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:28:42.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:488: TEST_just_deep_scrubs: sleep 5 2024-09-18T16:28:47.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:489: TEST_just_deep_scrubs: declare -A sc_data_2 2024-09-18T16:28:47.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:490: TEST_just_deep_scrubs: extract_published_sch 1.2 2024-09-18T16:28:41,169715147+00:00 2024-09-18T16:28:41,169715147+00:00 sc_data_2 2024-09-18T16:28:47.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.2 2024-09-18T16:28:47.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sc_data_2 2024-09-18T16:28:47.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2024-09-18T16:28:41,169715147+00:00 2024-09-18T16:28:47.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2024-09-18T16:28:41,169715147+00:00 2024-09-18T16:28:47.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=1 2024-09-18T16:28:47.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2024-09-18T16:28:47.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2024-09-18T16:28:47.521 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:28:47.701 INFO:tasks.workunit.client.0.smithi046.stdout:( 2024-09-18T16:28:47.701 INFO:tasks.workunit.client.0.smithi046.stdout:[query_epoch]=23 2024-09-18T16:28:47.701 INFO:tasks.workunit.client.0.smithi046.stdout:[query_seq]=16 2024-09-18T16:28:47.701 INFO:tasks.workunit.client.0.smithi046.stdout:[query_active]=false 2024-09-18T16:28:47.701 INFO:tasks.workunit.client.0.smithi046.stdout:[query_schedule]='queued for deep scrub' 2024-09-18T16:28:47.702 INFO:tasks.workunit.client.0.smithi046.stdout:[query_schedule_at]='0' 2024-09-18T16:28:47.702 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_duration]=0 2024-09-18T16:28:47.702 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_stamp]='2024-09-11T16:27:02.205419+0000' 2024-09-18T16:28:47.702 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_scrub]='0x0' 2024-09-18T16:28:47.702 INFO:tasks.workunit.client.0.smithi046.stdout:[query_is_future]=false 2024-09-18T16:28:47.702 INFO:tasks.workunit.client.0.smithi046.stdout:[query_vs_date]=false 2024-09-18T16:28:47.702 INFO:tasks.workunit.client.0.smithi046.stdout:[query_scrub_seq]=909312 2024-09-18T16:28:47.702 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_pg_state]='active+clean' 2024-09-18T16:28:47.702 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_state_has_scrubbing]=false 2024-09-18T16:28:47.702 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_last_duration]=0 2024-09-18T16:28:47.702 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_schedule]='periodic deep scrub scheduled' 2024-09-18T16:28:47.702 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_schedule_at]='2024-09-19T16:28:25.022552+0000' 2024-09-18T16:28:47.702 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_is_future]=true 2024-09-18T16:28:47.702 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_vs_date]=true 2024-09-18T16:28:47.703 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_reported_epoch]=23 2024-09-18T16:28:47.703 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_seq]=15 2024-09-18T16:28:47.703 INFO:tasks.workunit.client.0.smithi046.stdout:) 2024-09-18T16:28:47.747 INFO:tasks.workunit.client.0.smithi046.stdout:test counter @ should show no change: 909312 2024-09-18T16:28:47.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:491: TEST_just_deep_scrubs: echo 'test counter @ should show no change: ' 909312 2024-09-18T16:28:47.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:492: TEST_just_deep_scrubs: (( 0 == 0 )) 2024-09-18T16:28:47.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:493: TEST_just_deep_scrubs: (( 909312 == 909312 )) 2024-09-18T16:28:47.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:496: TEST_just_deep_scrubs: ceph osd unset noscrub 2024-09-18T16:28:48.336 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is unset 2024-09-18T16:28:48.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:497: TEST_just_deep_scrubs: sleep 5 2024-09-18T16:28:53.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:498: TEST_just_deep_scrubs: expct_qry_duration=(['query_last_duration']="0" ['query_last_duration_neg']="not0") 2024-09-18T16:28:53.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:498: TEST_just_deep_scrubs: declare -A expct_qry_duration 2024-09-18T16:28:53.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:499: TEST_just_deep_scrubs: sc_data_2=() 2024-09-18T16:28:53.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:500: TEST_just_deep_scrubs: echo 'test counter @ should be higher than before the unset: ' 2024-09-18T16:28:53.352 INFO:tasks.workunit.client.0.smithi046.stdout:test counter @ should be higher than before the unset: 2024-09-18T16:28:53.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:501: TEST_just_deep_scrubs: wait_any_cond 1.2 10 2024-09-18T16:28:25.022552+0000 expct_qry_duration 'WaitingAfterScrub ' sc_data_2 2024-09-18T16:28:53.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:98: wait_any_cond: local pgid=1.2 2024-09-18T16:28:53.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:99: wait_any_cond: local retries=10 2024-09-18T16:28:53.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:100: wait_any_cond: local cmp_date=2024-09-18T16:28:25.022552+0000 2024-09-18T16:28:53.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:101: wait_any_cond: local -n ep=expct_qry_duration 2024-09-18T16:28:53.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:102: wait_any_cond: local -n out_array=sc_data_2 2024-09-18T16:28:53.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:103: wait_any_cond: local -A sc_data 2024-09-18T16:28:53.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:104: wait_any_cond: local extr_dbg=2 2024-09-18T16:28:53.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local saved_echo_flag=x 2024-09-18T16:28:53.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: set +x 2024-09-18T16:28:53.354 INFO:tasks.workunit.client.0.smithi046.stdout:waiting for any condition (WaitingAfterScrub ): pg:1.2 dt:2024-09-18T16:28:25.022552+0000 (10 retries) 2024-09-18T16:28:54.155 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:28:54.339 INFO:tasks.workunit.client.0.smithi046.stdout:( 2024-09-18T16:28:54.339 INFO:tasks.workunit.client.0.smithi046.stdout:[query_epoch]=24 2024-09-18T16:28:54.339 INFO:tasks.workunit.client.0.smithi046.stdout:[query_seq]=27 2024-09-18T16:28:54.340 INFO:tasks.workunit.client.0.smithi046.stdout:[query_active]=false 2024-09-18T16:28:54.340 INFO:tasks.workunit.client.0.smithi046.stdout:[query_schedule]='scrub scheduled' 2024-09-18T16:28:54.340 INFO:tasks.workunit.client.0.smithi046.stdout:[query_schedule_at]='2024-09-19T16:28:51.517298+0000' 2024-09-18T16:28:54.340 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_duration]=1 2024-09-18T16:28:54.340 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_stamp]='2024-09-18T16:28:51.517298+0000' 2024-09-18T16:28:54.340 INFO:tasks.workunit.client.0.smithi046.stdout:[query_last_scrub]='0x0' 2024-09-18T16:28:54.340 INFO:tasks.workunit.client.0.smithi046.stdout:[query_is_future]=true 2024-09-18T16:28:54.340 INFO:tasks.workunit.client.0.smithi046.stdout:[query_vs_date]=true 2024-09-18T16:28:54.340 INFO:tasks.workunit.client.0.smithi046.stdout:[query_scrub_seq]=909314 2024-09-18T16:28:54.340 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_pg_state]='active+clean' 2024-09-18T16:28:54.340 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_state_has_scrubbing]=false 2024-09-18T16:28:54.340 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_last_duration]=0 2024-09-18T16:28:54.341 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_schedule]='periodic deep scrub scheduled' 2024-09-18T16:28:54.341 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_schedule_at]='2024-09-19T16:28:25.022552+0000' 2024-09-18T16:28:54.341 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_is_future]=true 2024-09-18T16:28:54.341 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_vs_date]=true 2024-09-18T16:28:54.341 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_reported_epoch]=23 2024-09-18T16:28:54.341 INFO:tasks.workunit.client.0.smithi046.stdout:[dmp_seq]=15 2024-09-18T16:28:54.341 INFO:tasks.workunit.client.0.smithi046.stdout:) 2024-09-18T16:28:54.389 INFO:tasks.workunit.client.0.smithi046.stdout:--> loop: 1 ~ false / 27 / 15 / true / 2024-09-18T16:28:51.517298+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-09-18T16:28:54.390 INFO:tasks.workunit.client.0.smithi046.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 1 2024-09-18T16:28:54.390 INFO:tasks.workunit.client.0.smithi046.stdout:WaitingAfterScrub - 'query_last_duration' actual value (1) matches expected (0) (negation: 1) 2024-09-18T16:28:54.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:149: wait_any_cond: return 0 2024-09-18T16:28:54.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: teardown td/osd-scrub-test 2024-09-18T16:28:54.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-09-18T16:28:54.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:28:54.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-18T16:28:54.391 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:28:54.391 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:28:54.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:28:54.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:28:54.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:28:54.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:28:54.504 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:28:54.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:28:54.506 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:28:54.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:28:54.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:28:54.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:28:54.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:28:54.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:28:54.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:28:54.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:28:54.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:28:54.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:28:54.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-09-18T16:28:54.536 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:28:54.536 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:28:54.536 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:28:54.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.562265 2024-09-18T16:28:54.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:28:54.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:28:54.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-09-18T16:28:54.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: setup td/osd-scrub-test 2024-09-18T16:28:54.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-test 2024-09-18T16:28:54.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-test 2024-09-18T16:28:54.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-09-18T16:28:54.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:28:54.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-18T16:28:54.540 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:28:54.540 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:28:54.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:28:54.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:28:54.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:28:54.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:28:54.545 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:28:54.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:28:54.547 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:28:54.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:28:54.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:28:54.550 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:28:54.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:28:54.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:28:54.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:28:54.552 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:28:54.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:28:54.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:28:54.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-09-18T16:28:54.558 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:28:54.558 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:28:54.558 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:28:54.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.562265 2024-09-18T16:28:54.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:28:54.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:28:54.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-test 2024-09-18T16:28:54.562 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T16:28:54.562 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:28:54.562 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:28:54.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.562265 2024-09-18T16:28:54.565 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T16:28:54.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T16:28:54.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T16:28:54.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-09-18T16:28:54.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: TEST_pg_dump_objects_scrubbed td/osd-scrub-test 2024-09-18T16:28:54.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:622: TEST_pg_dump_objects_scrubbed: local dir=td/osd-scrub-test 2024-09-18T16:28:54.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:623: TEST_pg_dump_objects_scrubbed: local poolname=test 2024-09-18T16:28:54.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:624: TEST_pg_dump_objects_scrubbed: local OSDS=3 2024-09-18T16:28:54.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:625: TEST_pg_dump_objects_scrubbed: local objects=15 2024-09-18T16:28:54.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:626: TEST_pg_dump_objects_scrubbed: local timeout=10 2024-09-18T16:28:54.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:628: TEST_pg_dump_objects_scrubbed: TESTDATA=testdata.562265 2024-09-18T16:28:54.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:630: TEST_pg_dump_objects_scrubbed: setup td/osd-scrub-test 2024-09-18T16:28:54.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-test 2024-09-18T16:28:54.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-test 2024-09-18T16:28:54.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-09-18T16:28:54.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:28:54.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-18T16:28:54.569 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:28:54.569 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:28:54.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:28:54.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:28:54.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:28:54.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:28:54.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:28:54.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:28:54.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:28:54.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:28:54.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:28:54.578 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:28:54.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:28:54.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:28:54.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:28:54.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:28:54.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:28:54.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:28:54.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-09-18T16:28:54.587 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:28:54.587 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:28:54.587 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:28:54.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.562265 2024-09-18T16:28:54.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:28:54.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:28:54.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-test 2024-09-18T16:28:54.594 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T16:28:54.595 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:28:54.595 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:28:54.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.562265 2024-09-18T16:28:54.595 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T16:28:54.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T16:28:54.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T16:28:54.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-09-18T16:28:54.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:631: TEST_pg_dump_objects_scrubbed: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2024-09-18T16:28:54.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-test 2024-09-18T16:28:54.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T16:28:54.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T16:28:54.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T16:28:54.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-test/a 2024-09-18T16:28:54.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2024-09-18T16:28:55.004 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T16:28:55.005 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:28:55.005 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:28:55.005 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:28:55.005 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:28:55.005 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:28:55.006 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:28:55.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-18T16:28:55.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T16:28:55.301 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T16:28:55.301 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:28:55.301 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:28:55.301 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T16:28:55.302 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T16:28:55.302 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:28:55.302 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:28:55.302 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:28:55.306 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:28:55.306 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:28:55.306 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:28:55.306 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.562265/ceph-mon.a.asok 2024-09-18T16:28:55.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:28:55.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.562265/ceph-mon.a.asok config get fsid 2024-09-18T16:28:55.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T16:28:55.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:28:55.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:28:55.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T16:28:55.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T16:28:55.411 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:28:55.411 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:28:55.411 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:28:55.412 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:28:55.412 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:28:55.412 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:28:55.412 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.562265/ceph-mon.a.asok 2024-09-18T16:28:55.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:28:55.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.562265/ceph-mon.a.asok config get mon_host 2024-09-18T16:28:55.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:632: TEST_pg_dump_objects_scrubbed: run_mgr td/osd-scrub-test x 2024-09-18T16:28:55.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-test 2024-09-18T16:28:55.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T16:28:55.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T16:28:55.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T16:28:55.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-test/x 2024-09-18T16:28:55.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T16:28:56.356 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T16:28:56.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:28:56.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:28:56.357 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:28:56.357 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:28:56.357 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:28:56.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:28:56.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:28:56.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:28:56.393 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:633: TEST_pg_dump_objects_scrubbed: expr 3 - 1 2024-09-18T16:28:56.394 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:633: TEST_pg_dump_objects_scrubbed: seq 0 2 2024-09-18T16:28:56.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:633: TEST_pg_dump_objects_scrubbed: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:28:56.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:635: TEST_pg_dump_objects_scrubbed: run_osd td/osd-scrub-test 0 2024-09-18T16:28:56.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-09-18T16:28:56.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:28:56.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T16:28:56.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:28:56.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/0 2024-09-18T16:28:56.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-18T16:28:56.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:28:56.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:28:56.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:28:56.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-09-18T16:28:56.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-09-18T16:28:56.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:28:56.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:28:56.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-18T16:28:56.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:28:56.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:28:56.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:28:56.398 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:28:56.398 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:28:56.398 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:28:56.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:28:56.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:28:56.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:28:56.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:28:56.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:28:56.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-18T16:28:56.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-18T16:28:56.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:28:56.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:28:56.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:28:56.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:28:56.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:28:56.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:28:56.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/0 2024-09-18T16:28:56.400 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:28:56.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a70ba827-75b8-41be-868f-f0d08c306f3f 2024-09-18T16:28:56.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a70ba827-75b8-41be-868f-f0d08c306f3f' 2024-09-18T16:28:56.402 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 a70ba827-75b8-41be-868f-f0d08c306f3f 2024-09-18T16:28:56.402 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:28:56.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDI/+pmJDHQGBAA7RkqP1oFXsarwfZF5zSIcw== 2024-09-18T16:28:56.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDI/+pmJDHQGBAA7RkqP1oFXsarwfZF5zSIcw=="}' 2024-09-18T16:28:56.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a70ba827-75b8-41be-868f-f0d08c306f3f -i td/osd-scrub-test/0/new.json 2024-09-18T16:28:56.711 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:28:56.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/0/new.json 2024-09-18T16:28:56.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDI/+pmJDHQGBAA7RkqP1oFXsarwfZF5zSIcw== --osd-uuid a70ba827-75b8-41be-868f-f0d08c306f3f 2024-09-18T16:28:56.757 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:28:56.748+0000 7f3066bd1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:28:56.800 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:28:56.792+0000 7f3066bd1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:28:56.802 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:28:56.796+0000 7f3066bd1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:28:56.803 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:28:56.796+0000 7f3066bd1540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:28:56.803 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:28:56.796+0000 7f3066bd1540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:28:56.803 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:28:56.796+0000 7f3066bd1540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:28:56.846 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:28:56.840+0000 7f3066bd1540 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-09-18T16:29:02.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-09-18T16:29:02.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:29:02.880 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T16:29:02.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T16:29:02.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:29:03.256 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T16:29:03.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T16:29:03.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:29:03.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:29:03.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:29:03.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:29:03.286 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:29:03.280+0000 7feba7626540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:29:03.288 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:29:03.280+0000 7feba7626540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:29:03.290 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:29:03.284+0000 7feba7626540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:29:03.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T16:29:03.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:29:03.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:29:03.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:29:03.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:29:03.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:03.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:29:03.550 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:29:03.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:03.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:29:03.836 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:29:03.828+0000 7feba7626540 -1 Falling back to public interface 2024-09-18T16:29:03.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:04.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:04.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:04.899 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:29:04.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:29:04.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:04.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:29:05.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:06.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:06.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:06.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:29:06.214 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:29:06.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:06.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:29:06.312 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:29:06.304+0000 7feba7626540 -1 osd.0 0 log_to_monitors true 2024-09-18T16:29:06.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:07.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:07.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:07.562 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:29:07.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:29:07.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:07.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:29:07.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:08.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:08.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:08.870 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:29:08.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:29:08.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:08.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:29:09.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:10.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:10.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:10.239 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:29:10.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:29:10.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:10.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:29:10.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:11.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:11.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:11.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:29:11.554 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:29:11.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:11.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:29:11.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:12.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:12.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:12.868 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:29:12.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:29:12.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:12.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:29:13.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:14.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:14.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:14.257 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:29:14.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:29:14.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:14.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:29:14.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:15.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:15.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:15.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:29:15.570 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:29:15.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:15.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:29:15.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:16.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:16.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:16.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:29:16.882 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:29:16.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:16.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:29:17.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:18.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:18.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:18.206 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:29:18.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:29:18.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:18.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:29:18.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:19.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:19.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:19.523 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:29:19.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:29:19.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:19.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:29:20.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:21.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:21.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:21.182 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:29:21.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:29:21.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:21.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:29:21.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:22.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:22.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:22.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:29:22.895 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:29:22.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:22.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:29:23.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:24.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:24.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:24.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:29:24.536 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:29:24.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:24.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:29:25.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:26.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:26.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:26.245 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:29:26.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:29:26.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:26.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:29:26.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:27.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:27.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:27.751 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:29:27.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:29:27.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:27.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:29:28.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:29.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:29.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:29.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:29:29.504 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:29:29.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:29.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:29:29.808 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/611244,v1:127.0.0.1:6803/611244] [v2:127.0.0.1:6804/611244,v1:127.0.0.1:6805/611244] exists,up a70ba827-75b8-41be-868f-f0d08c306f3f 2024-09-18T16:29:29.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:29:29.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:29:29.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:29:29.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:633: TEST_pg_dump_objects_scrubbed: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:29:29.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:635: TEST_pg_dump_objects_scrubbed: run_osd td/osd-scrub-test 1 2024-09-18T16:29:29.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-09-18T16:29:29.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:29:29.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T16:29:29.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:29:29.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/1 2024-09-18T16:29:29.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-18T16:29:29.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:29:29.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:29:29.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:29:29.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-09-18T16:29:29.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-09-18T16:29:29.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:29:29.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:29:29.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-18T16:29:29.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:29:29.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:29:29.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:29:29.814 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:29:29.814 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:29:29.814 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:29:29.814 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:29:29.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:29:29.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:29:29.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:29:29.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:29:29.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-18T16:29:29.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-18T16:29:29.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:29:29.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:29:29.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:29:29.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:29:29.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:29:29.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:29:29.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/1 2024-09-18T16:29:29.815 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:29:29.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=70f14103-13f5-477a-b01d-95f201903eca 2024-09-18T16:29:29.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 70f14103-13f5-477a-b01d-95f201903eca' 2024-09-18T16:29:29.817 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 70f14103-13f5-477a-b01d-95f201903eca 2024-09-18T16:29:29.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:29:29.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDp/+pm5W7QMRAA50jHCnKaPq9RI4K2wpcMUg== 2024-09-18T16:29:29.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDp/+pm5W7QMRAA50jHCnKaPq9RI4K2wpcMUg=="}' 2024-09-18T16:29:29.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 70f14103-13f5-477a-b01d-95f201903eca -i td/osd-scrub-test/1/new.json 2024-09-18T16:29:30.264 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:29:30.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/1/new.json 2024-09-18T16:29:30.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDp/+pm5W7QMRAA50jHCnKaPq9RI4K2wpcMUg== --osd-uuid 70f14103-13f5-477a-b01d-95f201903eca 2024-09-18T16:29:30.312 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:29:30.308+0000 7fa9662fc540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:29:30.325 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:29:30.320+0000 7fa9662fc540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:29:30.325 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:29:30.320+0000 7fa9662fc540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:29:30.325 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:29:30.320+0000 7fa9662fc540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:29:30.325 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:29:30.320+0000 7fa9662fc540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:29:30.326 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:29:30.320+0000 7fa9662fc540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:29:30.393 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:29:30.388+0000 7fa9662fc540 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-09-18T16:29:36.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-09-18T16:29:36.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:29:36.444 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T16:29:36.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T16:29:36.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:29:36.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T16:29:36.838 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T16:29:36.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:29:36.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:29:36.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:29:36.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:29:36.868 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:29:36.864+0000 7f6f83412540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:29:36.872 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:29:36.868+0000 7f6f83412540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:29:36.873 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:29:36.872+0000 7f6f83412540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:29:37.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T16:29:37.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:29:37.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T16:29:37.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:29:37.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:29:37.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:37.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:29:37.131 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:29:37.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:37.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:29:37.207 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:29:37.204+0000 7f6f83412540 -1 Falling back to public interface 2024-09-18T16:29:37.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:38.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:38.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:38.425 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:29:38.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:29:38.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:38.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:29:38.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:39.642 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:29:39.640+0000 7f6f83412540 -1 osd.1 0 log_to_monitors true 2024-09-18T16:29:39.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:39.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:39.790 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:29:39.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:29:39.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:39.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:29:40.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:41.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:41.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:41.169 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:29:41.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:29:41.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:41.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:29:41.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:42.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:42.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:42.484 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:29:42.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:29:42.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:42.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:29:42.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:43.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:43.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:43.927 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:29:43.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:29:43.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:29:43.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:44.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:45.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:45.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:45.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:29:45.253 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:29:45.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:45.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:29:45.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:46.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:46.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:46.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:29:46.606 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:29:46.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:46.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:29:46.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:47.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:47.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:47.928 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:29:47.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:29:47.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:47.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:29:48.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:49.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:49.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:49.243 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:29:49.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:29:49.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:49.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:29:49.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:50.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:50.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:50.598 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:29:50.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:29:50.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:50.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:29:50.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:51.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:51.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:51.956 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:29:51.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:29:51.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:51.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:29:52.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:53.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:53.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:53.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:29:53.405 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:29:53.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:53.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:29:54.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:55.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:55.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:55.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:29:55.315 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:29:55.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:55.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:29:55.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:56.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:56.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:56.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:29:56.952 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:29:56.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:56.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:29:57.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:58.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:58.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:58.263 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:29:58.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:29:58.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:58.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:29:58.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:29:59.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:29:59.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:29:59.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:29:59.929 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:29:59.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:29:59.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:30:00.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:30:01.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:30:01.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:30:01.484 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:30:01.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:30:01.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:30:01.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:30:01.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:30:02.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:30:02.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:30:02.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:30:02.920 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:30:02.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:30:02.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:30:03.232 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/613023,v1:127.0.0.1:6811/613023] [v2:127.0.0.1:6812/613023,v1:127.0.0.1:6813/613023] exists,up 70f14103-13f5-477a-b01d-95f201903eca 2024-09-18T16:30:03.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:30:03.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:30:03.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:30:03.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:633: TEST_pg_dump_objects_scrubbed: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:30:03.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:635: TEST_pg_dump_objects_scrubbed: run_osd td/osd-scrub-test 2 2024-09-18T16:30:03.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-09-18T16:30:03.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:30:03.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T16:30:03.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:30:03.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/2 2024-09-18T16:30:03.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-18T16:30:03.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:30:03.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:30:03.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:30:03.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2024-09-18T16:30:03.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2024-09-18T16:30:03.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:30:03.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:30:03.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-18T16:30:03.236 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:30:03.236 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:30:03.236 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:30:03.238 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:30:03.238 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:30:03.238 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:30:03.238 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:30:03.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:30:03.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:30:03.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:30:03.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:30:03.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-18T16:30:03.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-18T16:30:03.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:30:03.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:30:03.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:30:03.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:30:03.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:30:03.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:30:03.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/2 2024-09-18T16:30:03.239 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:30:03.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d791c7e5-078e-4afb-8b06-30ed015eb11a 2024-09-18T16:30:03.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d791c7e5-078e-4afb-8b06-30ed015eb11a' 2024-09-18T16:30:03.240 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 d791c7e5-078e-4afb-8b06-30ed015eb11a 2024-09-18T16:30:03.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:30:03.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQALAOtmO79BDxAAZgldIHr0oRJy65CDji6DGg== 2024-09-18T16:30:03.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQALAOtmO79BDxAAZgldIHr0oRJy65CDji6DGg=="}' 2024-09-18T16:30:03.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d791c7e5-078e-4afb-8b06-30ed015eb11a -i td/osd-scrub-test/2/new.json 2024-09-18T16:30:03.733 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:30:03.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/2/new.json 2024-09-18T16:30:03.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQALAOtmO79BDxAAZgldIHr0oRJy65CDji6DGg== --osd-uuid d791c7e5-078e-4afb-8b06-30ed015eb11a 2024-09-18T16:30:03.787 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:30:03.783+0000 7f7dc5317540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:30:03.791 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:30:03.787+0000 7f7dc5317540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:30:03.792 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:30:03.787+0000 7f7dc5317540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:30:03.794 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:30:03.791+0000 7f7dc5317540 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-09-18T16:30:03.794 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:30:03.791+0000 7f7dc5317540 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-09-18T16:30:03.794 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:30:03.791+0000 7f7dc5317540 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-09-18T16:30:03.875 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:30:03.871+0000 7f7dc5317540 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-09-18T16:30:10.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2024-09-18T16:30:10.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:30:10.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T16:30:10.070 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T16:30:10.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:30:10.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T16:30:10.463 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T16:30:10.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:30:10.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:30:10.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:30:10.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:30:10.490 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:30:10.487+0000 7f7192729540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:30:10.493 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:30:10.491+0000 7f7192729540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:30:10.496 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:30:10.491+0000 7f7192729540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:30:10.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T16:30:10.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:30:10.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T16:30:10.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:30:10.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:30:10.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:30:10.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:30:10.766 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:30:10.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:30:10.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:30:11.031 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:30:11.027+0000 7f7192729540 -1 Falling back to public interface 2024-09-18T16:30:11.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:30:12.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:30:12.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:30:12.114 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:30:12.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:30:12.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:30:12.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:30:12.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:30:13.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:30:13.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:30:13.423 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:30:13.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:30:13.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:30:13.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:30:13.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:30:13.764 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:30:13.759+0000 7f7192729540 -1 osd.2 0 log_to_monitors true 2024-09-18T16:30:14.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:30:14.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:30:14.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:30:14.732 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:30:14.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:30:14.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:30:15.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:30:16.035 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:30:16.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:30:16.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:30:16.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:30:16.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:30:16.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:30:16.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:30:17.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:30:17.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:30:17.347 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:30:17.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:30:17.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:30:17.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:30:17.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:30:18.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:30:18.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:30:18.675 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:30:18.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:30:18.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:30:18.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:30:18.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:30:19.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:30:19.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:30:19.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:30:19.982 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:30:19.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:30:19.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:30:20.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:30:21.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:30:21.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:30:21.293 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:30:21.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:30:21.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:30:21.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:30:21.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:30:22.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:30:22.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:30:22.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:30:22.707 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:30:22.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:30:22.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:30:23.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:30:24.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:30:24.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:30:24.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:30:24.021 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:30:24.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:30:24.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:30:24.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:30:25.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:30:25.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:30:25.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:30:25.333 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:30:25.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:30:25.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:30:26.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:30:27.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:30:27.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:30:27.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:30:27.350 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:30:27.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:30:27.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:30:28.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:30:29.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:30:29.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:30:29.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:30:29.252 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:30:29.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:30:29.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:30:29.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:30:30.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:30:30.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:30:30.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:30:30.860 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:30:30.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:30:30.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:30:31.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:30:32.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:30:32.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:30:32.179 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:30:32.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:30:32.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:30:32.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:30:32.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:30:33.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:30:33.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:30:33.606 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:30:33.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:30:33.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:30:33.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:30:33.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:30:34.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:30:34.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:30:34.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:30:34.991 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:30:34.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:30:34.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:30:35.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:30:36.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:30:36.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:30:36.715 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:30:36.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:30:36.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:30:36.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:30:37.017 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/614804,v1:127.0.0.1:6819/614804] [v2:127.0.0.1:6820/614804,v1:127.0.0.1:6821/614804] exists,up d791c7e5-078e-4afb-8b06-30ed015eb11a 2024-09-18T16:30:37.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:30:37.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:30:37.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:30:37.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:639: TEST_pg_dump_objects_scrubbed: create_pool test 1 1 2024-09-18T16:30:37.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-18T16:30:37.498 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'test' created 2024-09-18T16:30:37.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T16:30:38.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:640: TEST_pg_dump_objects_scrubbed: wait_for_clean 2024-09-18T16:30:38.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:30:38.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:30:38.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:30:38.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:30:38.517 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:30:38.518 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:30:38.518 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:30:38.518 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:30:38.518 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:30:38.518 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:30:38.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:30:38.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:30:38.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:30:38.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=500 2024-09-18T16:30:38.737 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:30:39.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:30:39.081 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:30:39.081 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:30:39.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:30:39.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:30:39.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:30:39.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836500 2024-09-18T16:30:39.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836500 2024-09-18T16:30:39.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500' 2024-09-18T16:30:39.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:30:39.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:30:39.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672977 2024-09-18T16:30:39.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672977 2024-09-18T16:30:39.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672977' 2024-09-18T16:30:39.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:30:39.399 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:30:39.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509450 2024-09-18T16:30:39.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509450 2024-09-18T16:30:39.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672977 2-64424509450' 2024-09-18T16:30:39.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:30:39.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836500 2024-09-18T16:30:39.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:30:39.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:30:39.542 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836500 2024-09-18T16:30:39.543 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:30:39.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836500 2024-09-18T16:30:39.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-18T16:30:39.544 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836500 2024-09-18T16:30:39.545 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:30:39.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836499 -lt 21474836500 2024-09-18T16:30:39.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:30:40.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-18T16:30:40.861 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:30:41.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836501 -lt 21474836500 2024-09-18T16:30:41.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:30:41.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672977 2024-09-18T16:30:41.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:30:41.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:30:41.174 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672977 2024-09-18T16:30:41.174 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:30:41.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672977 2024-09-18T16:30:41.176 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672977 2024-09-18T16:30:41.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-18T16:30:41.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:30:41.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-18T16:30:41.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:30:41.506 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509450 2024-09-18T16:30:41.506 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:30:41.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:30:41.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509450 2024-09-18T16:30:41.510 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:30:41.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509450 2024-09-18T16:30:41.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-09-18T16:30:41.512 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509450 2024-09-18T16:30:41.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:30:41.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509450 -lt 64424509450 2024-09-18T16:30:41.845 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:30:41.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:30:41.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:30:42.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:30:42.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:30:42.204 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:30:42.204 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:30:42.204 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:30:42.204 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:30:42.204 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:30:42.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:30:42.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:30:42.534 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:30:42.535 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:30:42.535 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:30:42.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:30:42.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:30:42.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:30:42.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:641: TEST_pg_dump_objects_scrubbed: ceph osd dump 2024-09-18T16:30:42.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:641: TEST_pg_dump_objects_scrubbed: grep '^pool.*['\'']test['\'']' 2024-09-18T16:30:42.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:641: TEST_pg_dump_objects_scrubbed: awk '{ print $2 }' 2024-09-18T16:30:43.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:641: TEST_pg_dump_objects_scrubbed: poolid=1 2024-09-18T16:30:43.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:643: TEST_pg_dump_objects_scrubbed: dd if=/dev/urandom of=testdata.562265 bs=1032 count=1 2024-09-18T16:30:43.237 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records in 2024-09-18T16:30:43.238 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records out 2024-09-18T16:30:43.238 INFO:tasks.workunit.client.0.smithi046.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 8.0352e-05 s, 12.8 MB/s 2024-09-18T16:30:43.238 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:644: TEST_pg_dump_objects_scrubbed: seq 1 15 2024-09-18T16:30:43.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:644: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-18T16:30:43.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:646: TEST_pg_dump_objects_scrubbed: rados -p test put obj1 testdata.562265 2024-09-18T16:30:43.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:644: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-18T16:30:43.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:646: TEST_pg_dump_objects_scrubbed: rados -p test put obj2 testdata.562265 2024-09-18T16:30:43.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:644: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-18T16:30:43.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:646: TEST_pg_dump_objects_scrubbed: rados -p test put obj3 testdata.562265 2024-09-18T16:30:43.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:644: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-18T16:30:43.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:646: TEST_pg_dump_objects_scrubbed: rados -p test put obj4 testdata.562265 2024-09-18T16:30:43.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:644: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-18T16:30:43.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:646: TEST_pg_dump_objects_scrubbed: rados -p test put obj5 testdata.562265 2024-09-18T16:30:44.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:644: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-18T16:30:44.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:646: TEST_pg_dump_objects_scrubbed: rados -p test put obj6 testdata.562265 2024-09-18T16:30:44.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:644: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-18T16:30:44.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:646: TEST_pg_dump_objects_scrubbed: rados -p test put obj7 testdata.562265 2024-09-18T16:30:44.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:644: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-18T16:30:44.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:646: TEST_pg_dump_objects_scrubbed: rados -p test put obj8 testdata.562265 2024-09-18T16:30:44.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:644: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-18T16:30:44.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:646: TEST_pg_dump_objects_scrubbed: rados -p test put obj9 testdata.562265 2024-09-18T16:30:44.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:644: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-18T16:30:44.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:646: TEST_pg_dump_objects_scrubbed: rados -p test put obj10 testdata.562265 2024-09-18T16:30:44.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:644: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-18T16:30:44.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:646: TEST_pg_dump_objects_scrubbed: rados -p test put obj11 testdata.562265 2024-09-18T16:30:45.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:644: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-18T16:30:45.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:646: TEST_pg_dump_objects_scrubbed: rados -p test put obj12 testdata.562265 2024-09-18T16:30:45.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:644: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-18T16:30:45.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:646: TEST_pg_dump_objects_scrubbed: rados -p test put obj13 testdata.562265 2024-09-18T16:30:45.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:644: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-18T16:30:45.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:646: TEST_pg_dump_objects_scrubbed: rados -p test put obj14 testdata.562265 2024-09-18T16:30:45.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:644: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-18T16:30:45.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:646: TEST_pg_dump_objects_scrubbed: rados -p test put obj15 testdata.562265 2024-09-18T16:30:45.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:648: TEST_pg_dump_objects_scrubbed: rm -f testdata.562265 2024-09-18T16:30:45.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:650: TEST_pg_dump_objects_scrubbed: local pgid=1.0 2024-09-18T16:30:45.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:652: TEST_pg_dump_objects_scrubbed: pg_scrub 1.0 2024-09-18T16:30:45.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: pg_scrub: local pgid=1.0 2024-09-18T16:30:45.696 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: pg_scrub: get_last_scrub_stamp 1.0 2024-09-18T16:30:45.696 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:30:45.696 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:30:45.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:30:45.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:30:46.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: pg_scrub: local last_scrub=2024-09-18T16:30:37.410067+0000 2024-09-18T16:30:46.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_scrub: ceph pg scrub 1.0 2024-09-18T16:30:46.287 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 1.0 on osd.1 to scrub 2024-09-18T16:30:46.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_scrub: wait_for_scrub 1.0 2024-09-18T16:30:37.410067+0000 2024-09-18T16:30:46.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-18T16:30:46.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T16:30:37.410067+0000 2024-09-18T16:30:46.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T16:30:46.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T16:30:46.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:30:46.296 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:30:46.296 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:30:46.296 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:30:46.297 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:30:46.297 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:30:46.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:30:37.410067+0000 '>' 2024-09-18T16:30:37.410067+0000 2024-09-18T16:30:46.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:30:47.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:30:47.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:30:47.596 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:30:47.596 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:30:47.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:30:47.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:30:47.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:30:47.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:30:37.410067+0000 '>' 2024-09-18T16:30:37.410067+0000 2024-09-18T16:30:47.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:30:48.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:30:48.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:30:48.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:30:48.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:30:48.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:30:48.952 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:30:48.952 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:30:49.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:30:37.410067+0000 '>' 2024-09-18T16:30:37.410067+0000 2024-09-18T16:30:49.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:30:50.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:30:50.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:30:50.265 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:30:50.265 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:30:50.265 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:30:50.265 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:30:50.266 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:30:50.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:30:47.227659+0000 '>' 2024-09-18T16:30:37.410067+0000 2024-09-18T16:30:50.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T16:30:50.578 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:653: TEST_pg_dump_objects_scrubbed: ceph pg 1.0 query 2024-09-18T16:30:50.578 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:653: TEST_pg_dump_objects_scrubbed: jq .info.stats.objects_scrubbed 2024-09-18T16:30:50.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:653: TEST_pg_dump_objects_scrubbed: test 15 = 15 2024-09-18T16:30:50.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:655: TEST_pg_dump_objects_scrubbed: teardown td/osd-scrub-test 2024-09-18T16:30:50.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-09-18T16:30:50.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:30:50.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-18T16:30:50.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:30:50.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:30:50.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:30:50.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:30:50.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:30:50.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:30:50.828 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:30:50.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:30:50.830 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:30:50.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:30:50.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:30:50.832 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:30:50.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:30:50.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:30:50.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:30:50.836 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:30:50.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:30:50.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:30:50.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-09-18T16:30:50.860 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:30:50.860 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:30:50.860 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:30:50.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.562265 2024-09-18T16:30:50.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:30:50.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:30:50.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: teardown td/osd-scrub-test 2024-09-18T16:30:50.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-09-18T16:30:50.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:30:50.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-18T16:30:50.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:30:50.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:30:50.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:30:50.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:30:50.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:30:50.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:30:50.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:30:50.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:30:50.871 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:30:50.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:30:50.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:30:50.874 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:30:50.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:30:50.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:30:50.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:30:50.876 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:30:50.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:30:50.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:30:50.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-09-18T16:30:50.882 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:30:50.882 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:30:50.882 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:30:50.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.562265 2024-09-18T16:30:50.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:30:50.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:30:50.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-09-18T16:30:50.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: setup td/osd-scrub-test 2024-09-18T16:30:50.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-test 2024-09-18T16:30:50.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-test 2024-09-18T16:30:50.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-09-18T16:30:50.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:30:50.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-18T16:30:50.886 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:30:50.886 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:30:50.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:30:50.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:30:50.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:30:50.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:30:50.890 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:30:50.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:30:50.892 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:30:50.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:30:50.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:30:50.895 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:30:50.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:30:50.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:30:50.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:30:50.897 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:30:50.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:30:50.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:30:50.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-09-18T16:30:50.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:30:50.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:30:50.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:30:50.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.562265 2024-09-18T16:30:50.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:30:50.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:30:50.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-test 2024-09-18T16:30:50.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T16:30:50.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:30:50.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:30:50.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.562265 2024-09-18T16:30:50.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T16:30:50.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T16:30:50.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T16:30:50.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-09-18T16:30:50.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: TEST_scrub_abort td/osd-scrub-test 2024-09-18T16:30:50.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:391: TEST_scrub_abort: local dir=td/osd-scrub-test 2024-09-18T16:30:50.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:392: TEST_scrub_abort: _scrub_abort td/osd-scrub-test scrub 2024-09-18T16:30:50.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:276: _scrub_abort: local dir=td/osd-scrub-test 2024-09-18T16:30:50.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:277: _scrub_abort: local poolname=test 2024-09-18T16:30:50.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:278: _scrub_abort: local OSDS=3 2024-09-18T16:30:50.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:279: _scrub_abort: local objects=1000 2024-09-18T16:30:50.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:280: _scrub_abort: local type=scrub 2024-09-18T16:30:50.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:282: _scrub_abort: TESTDATA=testdata.562265 2024-09-18T16:30:50.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:283: _scrub_abort: test scrub = scrub 2024-09-18T16:30:50.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:285: _scrub_abort: stopscrub=noscrub 2024-09-18T16:30:50.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:286: _scrub_abort: check=noscrub 2024-09-18T16:30:50.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:292: _scrub_abort: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2024-09-18T16:30:50.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-test 2024-09-18T16:30:50.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T16:30:50.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T16:30:50.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T16:30:50.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-test/a 2024-09-18T16:30:50.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2024-09-18T16:30:51.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T16:30:51.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:30:51.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:30:51.285 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:30:51.285 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:30:51.285 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:30:51.285 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:30:51.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-18T16:30:51.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T16:30:51.558 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T16:30:51.558 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:30:51.559 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:30:51.559 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T16:30:51.559 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:30:51.559 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:30:51.560 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:30:51.560 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T16:30:51.560 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:30:51.560 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:30:51.561 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:30:51.561 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.562265/ceph-mon.a.asok 2024-09-18T16:30:51.561 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:30:51.561 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.562265/ceph-mon.a.asok config get fsid 2024-09-18T16:30:51.654 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T16:30:51.654 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:30:51.654 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:30:51.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T16:30:51.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T16:30:51.655 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:30:51.655 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:30:51.655 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:30:51.656 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:30:51.656 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:30:51.656 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:30:51.656 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.562265/ceph-mon.a.asok 2024-09-18T16:30:51.657 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:30:51.657 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.562265/ceph-mon.a.asok config get mon_host 2024-09-18T16:30:51.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:293: _scrub_abort: run_mgr td/osd-scrub-test x 2024-09-18T16:30:51.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-test 2024-09-18T16:30:51.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T16:30:51.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T16:30:51.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T16:30:51.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-test/x 2024-09-18T16:30:51.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T16:30:52.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T16:30:52.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:30:52.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:30:52.603 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:30:52.603 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:30:52.603 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:30:52.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:30:52.604 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:30:52.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:30:52.643 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:294: _scrub_abort: expr 3 - 1 2024-09-18T16:30:52.644 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:294: _scrub_abort: seq 0 2 2024-09-18T16:30:52.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:294: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:30:52.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:300: _scrub_abort: run_osd td/osd-scrub-test 0 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-18T16:30:52.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-09-18T16:30:52.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:30:52.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T16:30:52.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:30:52.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/0 2024-09-18T16:30:52.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-18T16:30:52.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:30:52.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:30:52.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:30:52.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-09-18T16:30:52.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-09-18T16:30:52.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:30:52.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:30:52.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-18T16:30:52.650 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:30:52.650 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:30:52.651 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:30:52.651 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:30:52.651 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:30:52.651 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:30:52.651 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:30:52.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:30:52.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:30:52.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:30:52.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:30:52.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-18T16:30:52.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-18T16:30:52.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:30:52.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:30:52.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:30:52.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:30:52.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:30:52.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-09-18T16:30:52.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/0 2024-09-18T16:30:52.652 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:30:52.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8bc16f07-ffc6-4abe-bfa4-e46023b904fa 2024-09-18T16:30:52.653 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 8bc16f07-ffc6-4abe-bfa4-e46023b904fa 2024-09-18T16:30:52.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8bc16f07-ffc6-4abe-bfa4-e46023b904fa' 2024-09-18T16:30:52.654 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:30:52.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA8AOtmD9L/JxAAHQxjwe6mxtHzBAub4DcB3w== 2024-09-18T16:30:52.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA8AOtmD9L/JxAAHQxjwe6mxtHzBAub4DcB3w=="}' 2024-09-18T16:30:52.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8bc16f07-ffc6-4abe-bfa4-e46023b904fa -i td/osd-scrub-test/0/new.json 2024-09-18T16:30:52.965 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:30:52.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/0/new.json 2024-09-18T16:30:52.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQA8AOtmD9L/JxAAHQxjwe6mxtHzBAub4DcB3w== --osd-uuid 8bc16f07-ffc6-4abe-bfa4-e46023b904fa 2024-09-18T16:30:53.009 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:30:53.003+0000 7f2a4ef4e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:30:53.012 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:30:53.007+0000 7f2a4ef4e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:30:53.014 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:30:53.011+0000 7f2a4ef4e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:30:53.015 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:30:53.011+0000 7f2a4ef4e540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:30:53.015 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:30:53.011+0000 7f2a4ef4e540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:30:53.015 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:30:53.011+0000 7f2a4ef4e540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:30:53.081 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:30:53.079+0000 7f2a4ef4e540 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-09-18T16:30:58.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-09-18T16:30:58.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:30:58.826 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T16:30:58.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T16:30:58.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:30:59.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T16:30:59.214 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T16:30:59.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:30:59.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-18T16:30:59.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:30:59.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:30:59.245 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:30:59.243+0000 7f8d787cb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:30:59.269 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:30:59.267+0000 7f8d787cb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:30:59.271 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:30:59.267+0000 7f8d787cb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:30:59.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T16:30:59.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:30:59.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:30:59.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:30:59.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:30:59.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:30:59.555 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:30:59.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:30:59.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:30:59.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:30:59.824 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:30:59.819+0000 7f8d787cb540 -1 Falling back to public interface 2024-09-18T16:30:59.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:31:00.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:31:00.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:31:00.901 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:31:00.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:31:00.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:31:00.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:31:01.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:31:02.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:31:02.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:31:02.207 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:31:02.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:31:02.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:31:02.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:31:02.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:31:02.556 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:31:02.551+0000 7f8d787cb540 -1 osd.0 0 log_to_monitors true 2024-09-18T16:31:03.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:31:03.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:31:03.523 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:31:03.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:31:03.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:31:03.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:31:03.830 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/617815,v1:127.0.0.1:6803/617815] [v2:127.0.0.1:6804/617815,v1:127.0.0.1:6805/617815] exists,up 8bc16f07-ffc6-4abe-bfa4-e46023b904fa 2024-09-18T16:31:03.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:31:03.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:31:03.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:31:03.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:294: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:31:03.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:300: _scrub_abort: run_osd td/osd-scrub-test 1 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-18T16:31:03.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-09-18T16:31:03.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:31:03.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T16:31:03.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:31:03.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/1 2024-09-18T16:31:03.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-18T16:31:03.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:31:03.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:31:03.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:31:03.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-09-18T16:31:03.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-09-18T16:31:03.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:31:03.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:31:03.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-18T16:31:03.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:31:03.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:31:03.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:31:03.836 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:31:03.836 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:31:03.836 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:31:03.836 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:31:03.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:31:03.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:31:03.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:31:03.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:31:03.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-18T16:31:03.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-18T16:31:03.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:31:03.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:31:03.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:31:03.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:31:03.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:31:03.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-09-18T16:31:03.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/1 2024-09-18T16:31:03.837 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:31:03.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=96350947-b77e-4cd1-a430-c00e422a485c 2024-09-18T16:31:03.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 96350947-b77e-4cd1-a430-c00e422a485c' 2024-09-18T16:31:03.838 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 96350947-b77e-4cd1-a430-c00e422a485c 2024-09-18T16:31:03.839 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:31:03.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBHAOtmig7dMhAAORIxWa/ybR1MmeoxJIz0xA== 2024-09-18T16:31:03.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBHAOtmig7dMhAAORIxWa/ybR1MmeoxJIz0xA=="}' 2024-09-18T16:31:03.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 96350947-b77e-4cd1-a430-c00e422a485c -i td/osd-scrub-test/1/new.json 2024-09-18T16:31:04.241 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:31:04.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/1/new.json 2024-09-18T16:31:04.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQBHAOtmig7dMhAAORIxWa/ybR1MmeoxJIz0xA== --osd-uuid 96350947-b77e-4cd1-a430-c00e422a485c 2024-09-18T16:31:04.292 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:31:04.287+0000 7fe68a43f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:31:04.308 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:31:04.303+0000 7fe68a43f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:31:04.309 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:31:04.307+0000 7fe68a43f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:31:04.310 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:31:04.307+0000 7fe68a43f540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:31:04.310 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:31:04.307+0000 7fe68a43f540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:31:04.310 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:31:04.307+0000 7fe68a43f540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:31:04.376 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:31:04.371+0000 7fe68a43f540 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-09-18T16:31:10.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-09-18T16:31:10.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:31:10.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T16:31:10.608 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T16:31:10.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:31:11.010 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T16:31:11.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T16:31:11.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-18T16:31:11.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:31:11.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:31:11.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:31:11.041 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:31:11.039+0000 7f0d3545f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:31:11.044 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:31:11.039+0000 7f0d3545f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:31:11.045 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:31:11.043+0000 7f0d3545f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:31:11.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T16:31:11.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:31:11.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T16:31:11.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:31:11.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:31:11.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:31:11.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:31:11.352 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:31:11.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:31:11.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:31:11.587 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:31:11.583+0000 7f0d3545f540 -1 Falling back to public interface 2024-09-18T16:31:11.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:31:12.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:31:12.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:31:12.734 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:31:12.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:31:12.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:31:12.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:31:13.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:31:14.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:31:14.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:31:14.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:31:14.046 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:31:14.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:31:14.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:31:14.343 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:31:14.339+0000 7f0d3545f540 -1 osd.1 0 log_to_monitors true 2024-09-18T16:31:14.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:31:15.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:31:15.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:31:15.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:31:15.357 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:31:15.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:31:15.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:31:15.713 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/619214,v1:127.0.0.1:6811/619214] [v2:127.0.0.1:6812/619214,v1:127.0.0.1:6813/619214] exists,up 96350947-b77e-4cd1-a430-c00e422a485c 2024-09-18T16:31:15.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:31:15.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:31:15.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:31:15.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:294: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:31:15.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:300: _scrub_abort: run_osd td/osd-scrub-test 2 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-18T16:31:15.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-09-18T16:31:15.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:31:15.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T16:31:15.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:31:15.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/2 2024-09-18T16:31:15.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-18T16:31:15.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:31:15.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:31:15.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:31:15.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2024-09-18T16:31:15.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2024-09-18T16:31:15.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:31:15.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:31:15.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-18T16:31:15.717 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:31:15.717 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:31:15.717 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:31:15.717 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:31:15.717 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:31:15.717 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:31:15.718 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:31:15.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:31:15.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:31:15.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:31:15.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:31:15.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-18T16:31:15.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-18T16:31:15.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:31:15.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:31:15.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:31:15.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:31:15.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:31:15.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-09-18T16:31:15.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/2 2024-09-18T16:31:15.720 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:31:15.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3178350f-6f76-471c-a570-52dc9cf92d71 2024-09-18T16:31:15.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 3178350f-6f76-471c-a570-52dc9cf92d71' 2024-09-18T16:31:15.720 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 3178350f-6f76-471c-a570-52dc9cf92d71 2024-09-18T16:31:15.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:31:15.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBTAOtmufXkKxAAfWoOUD1hwM36Z3QJaLOnBA== 2024-09-18T16:31:15.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBTAOtmufXkKxAAfWoOUD1hwM36Z3QJaLOnBA=="}' 2024-09-18T16:31:15.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3178350f-6f76-471c-a570-52dc9cf92d71 -i td/osd-scrub-test/2/new.json 2024-09-18T16:31:16.146 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:31:16.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/2/new.json 2024-09-18T16:31:16.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQBTAOtmufXkKxAAfWoOUD1hwM36Z3QJaLOnBA== --osd-uuid 3178350f-6f76-471c-a570-52dc9cf92d71 2024-09-18T16:31:16.200 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:31:16.195+0000 7f03d3c64540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:31:16.204 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:31:16.199+0000 7f03d3c64540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:31:16.206 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:31:16.203+0000 7f03d3c64540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:31:16.208 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:31:16.203+0000 7f03d3c64540 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-09-18T16:31:16.208 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:31:16.203+0000 7f03d3c64540 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-09-18T16:31:16.208 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:31:16.203+0000 7f03d3c64540 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-09-18T16:31:16.265 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:31:16.259+0000 7f03d3c64540 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-09-18T16:31:22.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2024-09-18T16:31:22.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:31:22.456 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T16:31:22.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T16:31:22.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:31:22.836 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T16:31:22.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T16:31:22.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-18T16:31:22.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:31:22.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:31:22.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:31:22.869 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:31:22.867+0000 7fbd73492540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:31:22.875 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:31:22.871+0000 7fbd73492540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:31:22.876 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:31:22.871+0000 7fbd73492540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:31:23.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T16:31:23.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:31:23.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T16:31:23.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:31:23.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:31:23.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:31:23.151 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:31:23.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:31:23.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:31:23.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:31:23.435 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:31:23.431+0000 7fbd73492540 -1 Falling back to public interface 2024-09-18T16:31:23.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:31:24.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:31:24.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:31:24.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:31:24.451 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:31:24.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:31:24.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:31:24.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:31:25.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:31:25.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:31:25.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:31:25.772 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:31:25.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:31:25.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:31:26.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:31:26.312 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:31:26.306+0000 7fbd73492540 -1 osd.2 0 log_to_monitors true 2024-09-18T16:31:27.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:31:27.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:31:27.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:31:27.094 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:31:27.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:31:27.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:31:27.448 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/620613,v1:127.0.0.1:6819/620613] [v2:127.0.0.1:6820/620613,v1:127.0.0.1:6821/620613] exists,up 3178350f-6f76-471c-a570-52dc9cf92d71 2024-09-18T16:31:27.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:31:27.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:31:27.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:31:27.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:308: _scrub_abort: create_pool test 1 1 2024-09-18T16:31:27.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-18T16:31:27.912 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'test' created 2024-09-18T16:31:27.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T16:31:28.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:309: _scrub_abort: wait_for_clean 2024-09-18T16:31:28.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:31:28.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:31:28.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:31:28.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:31:28.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:31:28.931 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:31:28.931 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:31:28.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:31:28.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:31:28.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:31:29.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:31:29.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:31:29.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:31:29.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=500 2024-09-18T16:31:29.162 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:31:29.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:31:29.488 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:31:29.489 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:31:29.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:31:29.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:31:29.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:31:29.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836487 2024-09-18T16:31:29.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836487 2024-09-18T16:31:29.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836487' 2024-09-18T16:31:29.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:31:29.667 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:31:29.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672964 2024-09-18T16:31:29.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672964 2024-09-18T16:31:29.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836487 1-42949672964' 2024-09-18T16:31:29.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:31:29.801 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:31:29.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509442 2024-09-18T16:31:29.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509442 2024-09-18T16:31:29.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836487 1-42949672964 2-64424509442' 2024-09-18T16:31:29.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:31:29.937 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836487 2024-09-18T16:31:29.937 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:31:29.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:31:29.940 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836487 2024-09-18T16:31:29.940 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:31:29.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836487 2024-09-18T16:31:29.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-09-18T16:31:29.943 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836487 2024-09-18T16:31:29.943 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:31:30.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836487 -lt 21474836487 2024-09-18T16:31:30.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:31:30.294 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672964 2024-09-18T16:31:30.295 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:31:30.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:31:30.298 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672964 2024-09-18T16:31:30.298 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:31:30.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672964 2024-09-18T16:31:30.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-09-18T16:31:30.299 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672964 2024-09-18T16:31:30.300 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:31:30.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672964 -lt 42949672964 2024-09-18T16:31:30.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:31:30.639 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509442 2024-09-18T16:31:30.639 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:31:30.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:31:30.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509442 2024-09-18T16:31:30.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:31:30.644 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509442 2024-09-18T16:31:30.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509442 2024-09-18T16:31:30.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-09-18T16:31:30.645 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:31:30.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509442 -lt 64424509442 2024-09-18T16:31:30.988 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:31:30.989 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:31:30.989 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:31:31.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:31:31.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:31:31.403 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:31:31.403 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:31:31.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:31:31.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:31:31.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:31:31.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:31:31.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T16:31:31.701 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:31:31.702 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:31:31.702 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:31:32.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T16:31:32.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-18T16:31:32.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T16:31:32.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-18T16:31:32.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T16:31:32.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T16:31:32.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T16:31:32.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:31:32.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:31:32.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:31:32.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:31:32.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:31:32.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:31:32.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:31:32.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:31:32.457 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:31:32.458 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:31:32.458 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:31:32.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:31:32.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:31:32.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:31:32.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:310: _scrub_abort: ceph osd dump 2024-09-18T16:31:32.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:310: _scrub_abort: grep '^pool.*['\'']test['\'']' 2024-09-18T16:31:32.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:310: _scrub_abort: awk '{ print $2 }' 2024-09-18T16:31:33.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:310: _scrub_abort: poolid=1 2024-09-18T16:31:33.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:312: _scrub_abort: dd if=/dev/urandom of=testdata.562265 bs=1032 count=1 2024-09-18T16:31:33.148 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records in 2024-09-18T16:31:33.149 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records out 2024-09-18T16:31:33.149 INFO:tasks.workunit.client.0.smithi046.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 0.000191524 s, 5.4 MB/s 2024-09-18T16:31:33.149 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: seq 1 1000 2024-09-18T16:31:33.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:33.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj1 testdata.562265 2024-09-18T16:31:33.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:33.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj2 testdata.562265 2024-09-18T16:31:33.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:33.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj3 testdata.562265 2024-09-18T16:31:33.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:33.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj4 testdata.562265 2024-09-18T16:31:33.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:33.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj5 testdata.562265 2024-09-18T16:31:34.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:34.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj6 testdata.562265 2024-09-18T16:31:34.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:34.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj7 testdata.562265 2024-09-18T16:31:34.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:34.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj8 testdata.562265 2024-09-18T16:31:34.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:34.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj9 testdata.562265 2024-09-18T16:31:34.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:34.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj10 testdata.562265 2024-09-18T16:31:34.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:34.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj11 testdata.562265 2024-09-18T16:31:34.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:34.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj12 testdata.562265 2024-09-18T16:31:35.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:35.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj13 testdata.562265 2024-09-18T16:31:35.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:35.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj14 testdata.562265 2024-09-18T16:31:35.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:35.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj15 testdata.562265 2024-09-18T16:31:35.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:35.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj16 testdata.562265 2024-09-18T16:31:35.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:35.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj17 testdata.562265 2024-09-18T16:31:35.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:35.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj18 testdata.562265 2024-09-18T16:31:36.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:36.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj19 testdata.562265 2024-09-18T16:31:36.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:36.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj20 testdata.562265 2024-09-18T16:31:36.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:36.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj21 testdata.562265 2024-09-18T16:31:36.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:36.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj22 testdata.562265 2024-09-18T16:31:36.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:36.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj23 testdata.562265 2024-09-18T16:31:36.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:36.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj24 testdata.562265 2024-09-18T16:31:37.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:37.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj25 testdata.562265 2024-09-18T16:31:37.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:37.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj26 testdata.562265 2024-09-18T16:31:37.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:37.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj27 testdata.562265 2024-09-18T16:31:37.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:37.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj28 testdata.562265 2024-09-18T16:31:37.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:37.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj29 testdata.562265 2024-09-18T16:31:37.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:37.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj30 testdata.562265 2024-09-18T16:31:37.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:37.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj31 testdata.562265 2024-09-18T16:31:38.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:38.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj32 testdata.562265 2024-09-18T16:31:38.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:38.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj33 testdata.562265 2024-09-18T16:31:38.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:38.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj34 testdata.562265 2024-09-18T16:31:38.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:38.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj35 testdata.562265 2024-09-18T16:31:38.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:38.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj36 testdata.562265 2024-09-18T16:31:38.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:38.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj37 testdata.562265 2024-09-18T16:31:39.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:39.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj38 testdata.562265 2024-09-18T16:31:39.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:39.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj39 testdata.562265 2024-09-18T16:31:39.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:39.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj40 testdata.562265 2024-09-18T16:31:39.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:39.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj41 testdata.562265 2024-09-18T16:31:39.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:39.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj42 testdata.562265 2024-09-18T16:31:39.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:39.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj43 testdata.562265 2024-09-18T16:31:39.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:39.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj44 testdata.562265 2024-09-18T16:31:40.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:40.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj45 testdata.562265 2024-09-18T16:31:40.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:40.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj46 testdata.562265 2024-09-18T16:31:40.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:40.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj47 testdata.562265 2024-09-18T16:31:40.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:40.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj48 testdata.562265 2024-09-18T16:31:40.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:40.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj49 testdata.562265 2024-09-18T16:31:40.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:40.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj50 testdata.562265 2024-09-18T16:31:41.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:41.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj51 testdata.562265 2024-09-18T16:31:41.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:41.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj52 testdata.562265 2024-09-18T16:31:41.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:41.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj53 testdata.562265 2024-09-18T16:31:41.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:41.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj54 testdata.562265 2024-09-18T16:31:41.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:41.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj55 testdata.562265 2024-09-18T16:31:41.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:41.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj56 testdata.562265 2024-09-18T16:31:41.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:41.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj57 testdata.562265 2024-09-18T16:31:42.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:42.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj58 testdata.562265 2024-09-18T16:31:42.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:42.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj59 testdata.562265 2024-09-18T16:31:42.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:42.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj60 testdata.562265 2024-09-18T16:31:42.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:42.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj61 testdata.562265 2024-09-18T16:31:42.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:42.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj62 testdata.562265 2024-09-18T16:31:42.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:42.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj63 testdata.562265 2024-09-18T16:31:43.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:43.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj64 testdata.562265 2024-09-18T16:31:43.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:43.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj65 testdata.562265 2024-09-18T16:31:43.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:43.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj66 testdata.562265 2024-09-18T16:31:43.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:43.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj67 testdata.562265 2024-09-18T16:31:43.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:43.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj68 testdata.562265 2024-09-18T16:31:43.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:43.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj69 testdata.562265 2024-09-18T16:31:43.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:43.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj70 testdata.562265 2024-09-18T16:31:44.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:44.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj71 testdata.562265 2024-09-18T16:31:44.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:44.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj72 testdata.562265 2024-09-18T16:31:44.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:44.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj73 testdata.562265 2024-09-18T16:31:44.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:44.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj74 testdata.562265 2024-09-18T16:31:44.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:44.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj75 testdata.562265 2024-09-18T16:31:44.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:44.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj76 testdata.562265 2024-09-18T16:31:45.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:45.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj77 testdata.562265 2024-09-18T16:31:45.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:45.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj78 testdata.562265 2024-09-18T16:31:45.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:45.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj79 testdata.562265 2024-09-18T16:31:45.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:45.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj80 testdata.562265 2024-09-18T16:31:45.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:45.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj81 testdata.562265 2024-09-18T16:31:45.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:45.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj82 testdata.562265 2024-09-18T16:31:45.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:45.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj83 testdata.562265 2024-09-18T16:31:46.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:46.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj84 testdata.562265 2024-09-18T16:31:46.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:46.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj85 testdata.562265 2024-09-18T16:31:46.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:46.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj86 testdata.562265 2024-09-18T16:31:46.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:46.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj87 testdata.562265 2024-09-18T16:31:46.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:46.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj88 testdata.562265 2024-09-18T16:31:46.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:46.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj89 testdata.562265 2024-09-18T16:31:46.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:46.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj90 testdata.562265 2024-09-18T16:31:47.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:47.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj91 testdata.562265 2024-09-18T16:31:47.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:47.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj92 testdata.562265 2024-09-18T16:31:47.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:47.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj93 testdata.562265 2024-09-18T16:31:47.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:47.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj94 testdata.562265 2024-09-18T16:31:47.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:47.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj95 testdata.562265 2024-09-18T16:31:47.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:47.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj96 testdata.562265 2024-09-18T16:31:48.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:48.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj97 testdata.562265 2024-09-18T16:31:48.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:48.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj98 testdata.562265 2024-09-18T16:31:48.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:48.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj99 testdata.562265 2024-09-18T16:31:48.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:48.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj100 testdata.562265 2024-09-18T16:31:48.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:48.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj101 testdata.562265 2024-09-18T16:31:48.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:48.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj102 testdata.562265 2024-09-18T16:31:49.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:49.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj103 testdata.562265 2024-09-18T16:31:49.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:49.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj104 testdata.562265 2024-09-18T16:31:49.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:49.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj105 testdata.562265 2024-09-18T16:31:49.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:49.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj106 testdata.562265 2024-09-18T16:31:49.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:49.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj107 testdata.562265 2024-09-18T16:31:49.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:49.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj108 testdata.562265 2024-09-18T16:31:49.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:49.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj109 testdata.562265 2024-09-18T16:31:50.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:50.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj110 testdata.562265 2024-09-18T16:31:50.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:50.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj111 testdata.562265 2024-09-18T16:31:50.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:50.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj112 testdata.562265 2024-09-18T16:31:50.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:50.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj113 testdata.562265 2024-09-18T16:31:50.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:50.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj114 testdata.562265 2024-09-18T16:31:50.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:50.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj115 testdata.562265 2024-09-18T16:31:50.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:50.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj116 testdata.562265 2024-09-18T16:31:51.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:51.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj117 testdata.562265 2024-09-18T16:31:51.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:51.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj118 testdata.562265 2024-09-18T16:31:51.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:51.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj119 testdata.562265 2024-09-18T16:31:51.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:51.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj120 testdata.562265 2024-09-18T16:31:51.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:51.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj121 testdata.562265 2024-09-18T16:31:51.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:51.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj122 testdata.562265 2024-09-18T16:31:51.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:51.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj123 testdata.562265 2024-09-18T16:31:52.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:52.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj124 testdata.562265 2024-09-18T16:31:52.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:52.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj125 testdata.562265 2024-09-18T16:31:52.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:52.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj126 testdata.562265 2024-09-18T16:31:52.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:52.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj127 testdata.562265 2024-09-18T16:31:52.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:52.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj128 testdata.562265 2024-09-18T16:31:52.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:52.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj129 testdata.562265 2024-09-18T16:31:52.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:52.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj130 testdata.562265 2024-09-18T16:31:53.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:53.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj131 testdata.562265 2024-09-18T16:31:53.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:53.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj132 testdata.562265 2024-09-18T16:31:53.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:53.427 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj133 testdata.562265 2024-09-18T16:31:53.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:53.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj134 testdata.562265 2024-09-18T16:31:53.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:53.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj135 testdata.562265 2024-09-18T16:31:53.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:53.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj136 testdata.562265 2024-09-18T16:31:54.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:54.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj137 testdata.562265 2024-09-18T16:31:54.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:54.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj138 testdata.562265 2024-09-18T16:31:54.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:54.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj139 testdata.562265 2024-09-18T16:31:54.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:54.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj140 testdata.562265 2024-09-18T16:31:54.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:54.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj141 testdata.562265 2024-09-18T16:31:55.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:55.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj142 testdata.562265 2024-09-18T16:31:55.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:55.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj143 testdata.562265 2024-09-18T16:31:55.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:55.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj144 testdata.562265 2024-09-18T16:31:55.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:55.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj145 testdata.562265 2024-09-18T16:31:55.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:55.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj146 testdata.562265 2024-09-18T16:31:55.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:55.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj147 testdata.562265 2024-09-18T16:31:55.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:55.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj148 testdata.562265 2024-09-18T16:31:56.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:56.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj149 testdata.562265 2024-09-18T16:31:56.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:56.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj150 testdata.562265 2024-09-18T16:31:56.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:56.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj151 testdata.562265 2024-09-18T16:31:56.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:56.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj152 testdata.562265 2024-09-18T16:31:56.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:56.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj153 testdata.562265 2024-09-18T16:31:56.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:56.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj154 testdata.562265 2024-09-18T16:31:56.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:56.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj155 testdata.562265 2024-09-18T16:31:57.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:57.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj156 testdata.562265 2024-09-18T16:31:57.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:57.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj157 testdata.562265 2024-09-18T16:31:57.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:57.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj158 testdata.562265 2024-09-18T16:31:57.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:57.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj159 testdata.562265 2024-09-18T16:31:57.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:57.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj160 testdata.562265 2024-09-18T16:31:57.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:57.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj161 testdata.562265 2024-09-18T16:31:58.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:58.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj162 testdata.562265 2024-09-18T16:31:58.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:58.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj163 testdata.562265 2024-09-18T16:31:58.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:58.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj164 testdata.562265 2024-09-18T16:31:58.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:58.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj165 testdata.562265 2024-09-18T16:31:58.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:58.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj166 testdata.562265 2024-09-18T16:31:58.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:58.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj167 testdata.562265 2024-09-18T16:31:58.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:58.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj168 testdata.562265 2024-09-18T16:31:59.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:59.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj169 testdata.562265 2024-09-18T16:31:59.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:59.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj170 testdata.562265 2024-09-18T16:31:59.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:59.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj171 testdata.562265 2024-09-18T16:31:59.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:59.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj172 testdata.562265 2024-09-18T16:31:59.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:59.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj173 testdata.562265 2024-09-18T16:31:59.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:59.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj174 testdata.562265 2024-09-18T16:31:59.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:31:59.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj175 testdata.562265 2024-09-18T16:32:00.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:00.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj176 testdata.562265 2024-09-18T16:32:00.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:00.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj177 testdata.562265 2024-09-18T16:32:00.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:00.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj178 testdata.562265 2024-09-18T16:32:00.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:00.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj179 testdata.562265 2024-09-18T16:32:00.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:00.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj180 testdata.562265 2024-09-18T16:32:00.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:00.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj181 testdata.562265 2024-09-18T16:32:00.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:00.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj182 testdata.562265 2024-09-18T16:32:01.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:01.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj183 testdata.562265 2024-09-18T16:32:01.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:01.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj184 testdata.562265 2024-09-18T16:32:01.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:01.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj185 testdata.562265 2024-09-18T16:32:01.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:01.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj186 testdata.562265 2024-09-18T16:32:01.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:01.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj187 testdata.562265 2024-09-18T16:32:01.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:01.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj188 testdata.562265 2024-09-18T16:32:02.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:02.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj189 testdata.562265 2024-09-18T16:32:02.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:02.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj190 testdata.562265 2024-09-18T16:32:02.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:02.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj191 testdata.562265 2024-09-18T16:32:02.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:02.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj192 testdata.562265 2024-09-18T16:32:02.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:02.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj193 testdata.562265 2024-09-18T16:32:02.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:02.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj194 testdata.562265 2024-09-18T16:32:03.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:03.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj195 testdata.562265 2024-09-18T16:32:03.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:03.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj196 testdata.562265 2024-09-18T16:32:03.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:03.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj197 testdata.562265 2024-09-18T16:32:03.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:03.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj198 testdata.562265 2024-09-18T16:32:03.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:03.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj199 testdata.562265 2024-09-18T16:32:03.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:03.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj200 testdata.562265 2024-09-18T16:32:03.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:03.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj201 testdata.562265 2024-09-18T16:32:04.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:04.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj202 testdata.562265 2024-09-18T16:32:04.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:04.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj203 testdata.562265 2024-09-18T16:32:04.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:04.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj204 testdata.562265 2024-09-18T16:32:04.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:04.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj205 testdata.562265 2024-09-18T16:32:04.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:04.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj206 testdata.562265 2024-09-18T16:32:04.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:04.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj207 testdata.562265 2024-09-18T16:32:04.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:05.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj208 testdata.562265 2024-09-18T16:32:05.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:05.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj209 testdata.562265 2024-09-18T16:32:05.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:05.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj210 testdata.562265 2024-09-18T16:32:05.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:05.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj211 testdata.562265 2024-09-18T16:32:05.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:05.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj212 testdata.562265 2024-09-18T16:32:05.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:05.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj213 testdata.562265 2024-09-18T16:32:05.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:05.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj214 testdata.562265 2024-09-18T16:32:05.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:05.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj215 testdata.562265 2024-09-18T16:32:06.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:06.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj216 testdata.562265 2024-09-18T16:32:06.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:06.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj217 testdata.562265 2024-09-18T16:32:06.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:06.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj218 testdata.562265 2024-09-18T16:32:06.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:06.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj219 testdata.562265 2024-09-18T16:32:06.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:06.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj220 testdata.562265 2024-09-18T16:32:06.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:06.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj221 testdata.562265 2024-09-18T16:32:07.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:07.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj222 testdata.562265 2024-09-18T16:32:07.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:07.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj223 testdata.562265 2024-09-18T16:32:07.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:07.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj224 testdata.562265 2024-09-18T16:32:07.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:07.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj225 testdata.562265 2024-09-18T16:32:07.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:07.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj226 testdata.562265 2024-09-18T16:32:07.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:07.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj227 testdata.562265 2024-09-18T16:32:08.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:08.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj228 testdata.562265 2024-09-18T16:32:08.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:08.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj229 testdata.562265 2024-09-18T16:32:08.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:08.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj230 testdata.562265 2024-09-18T16:32:08.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:08.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj231 testdata.562265 2024-09-18T16:32:08.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:08.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj232 testdata.562265 2024-09-18T16:32:08.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:08.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj233 testdata.562265 2024-09-18T16:32:08.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:08.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj234 testdata.562265 2024-09-18T16:32:09.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:09.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj235 testdata.562265 2024-09-18T16:32:09.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:09.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj236 testdata.562265 2024-09-18T16:32:09.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:09.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj237 testdata.562265 2024-09-18T16:32:09.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:09.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj238 testdata.562265 2024-09-18T16:32:09.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:09.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj239 testdata.562265 2024-09-18T16:32:10.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:10.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj240 testdata.562265 2024-09-18T16:32:10.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:10.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj241 testdata.562265 2024-09-18T16:32:10.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:10.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj242 testdata.562265 2024-09-18T16:32:10.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:10.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj243 testdata.562265 2024-09-18T16:32:10.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:10.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj244 testdata.562265 2024-09-18T16:32:10.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:10.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj245 testdata.562265 2024-09-18T16:32:10.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:10.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj246 testdata.562265 2024-09-18T16:32:11.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:11.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj247 testdata.562265 2024-09-18T16:32:11.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:11.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj248 testdata.562265 2024-09-18T16:32:11.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:11.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj249 testdata.562265 2024-09-18T16:32:11.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:11.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj250 testdata.562265 2024-09-18T16:32:11.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:11.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj251 testdata.562265 2024-09-18T16:32:11.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:11.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj252 testdata.562265 2024-09-18T16:32:11.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:11.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj253 testdata.562265 2024-09-18T16:32:12.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:12.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj254 testdata.562265 2024-09-18T16:32:12.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:12.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj255 testdata.562265 2024-09-18T16:32:12.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:12.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj256 testdata.562265 2024-09-18T16:32:12.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:12.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj257 testdata.562265 2024-09-18T16:32:12.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:12.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj258 testdata.562265 2024-09-18T16:32:12.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:12.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj259 testdata.562265 2024-09-18T16:32:13.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:13.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj260 testdata.562265 2024-09-18T16:32:13.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:13.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj261 testdata.562265 2024-09-18T16:32:13.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:13.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj262 testdata.562265 2024-09-18T16:32:13.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:13.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj263 testdata.562265 2024-09-18T16:32:13.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:13.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj264 testdata.562265 2024-09-18T16:32:13.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:13.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj265 testdata.562265 2024-09-18T16:32:13.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:13.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj266 testdata.562265 2024-09-18T16:32:14.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:14.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj267 testdata.562265 2024-09-18T16:32:14.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:14.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj268 testdata.562265 2024-09-18T16:32:14.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:14.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj269 testdata.562265 2024-09-18T16:32:14.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:14.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj270 testdata.562265 2024-09-18T16:32:14.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:14.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj271 testdata.562265 2024-09-18T16:32:14.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:14.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj272 testdata.562265 2024-09-18T16:32:14.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:14.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj273 testdata.562265 2024-09-18T16:32:15.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:15.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj274 testdata.562265 2024-09-18T16:32:15.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:15.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj275 testdata.562265 2024-09-18T16:32:15.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:15.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj276 testdata.562265 2024-09-18T16:32:15.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:15.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj277 testdata.562265 2024-09-18T16:32:15.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:15.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj278 testdata.562265 2024-09-18T16:32:15.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:15.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj279 testdata.562265 2024-09-18T16:32:16.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:16.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj280 testdata.562265 2024-09-18T16:32:16.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:16.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj281 testdata.562265 2024-09-18T16:32:16.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:16.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj282 testdata.562265 2024-09-18T16:32:16.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:16.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj283 testdata.562265 2024-09-18T16:32:16.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:16.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj284 testdata.562265 2024-09-18T16:32:16.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:16.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj285 testdata.562265 2024-09-18T16:32:16.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:16.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj286 testdata.562265 2024-09-18T16:32:17.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:17.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj287 testdata.562265 2024-09-18T16:32:17.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:17.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj288 testdata.562265 2024-09-18T16:32:17.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:17.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj289 testdata.562265 2024-09-18T16:32:17.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:17.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj290 testdata.562265 2024-09-18T16:32:17.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:17.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj291 testdata.562265 2024-09-18T16:32:17.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:17.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj292 testdata.562265 2024-09-18T16:32:17.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:17.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj293 testdata.562265 2024-09-18T16:32:18.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:18.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj294 testdata.562265 2024-09-18T16:32:18.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:18.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj295 testdata.562265 2024-09-18T16:32:18.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:18.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj296 testdata.562265 2024-09-18T16:32:18.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:18.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj297 testdata.562265 2024-09-18T16:32:18.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:18.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj298 testdata.562265 2024-09-18T16:32:18.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:18.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj299 testdata.562265 2024-09-18T16:32:19.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:19.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj300 testdata.562265 2024-09-18T16:32:19.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:19.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj301 testdata.562265 2024-09-18T16:32:19.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:19.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj302 testdata.562265 2024-09-18T16:32:19.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:19.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj303 testdata.562265 2024-09-18T16:32:19.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:19.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj304 testdata.562265 2024-09-18T16:32:19.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:19.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj305 testdata.562265 2024-09-18T16:32:20.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:20.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj306 testdata.562265 2024-09-18T16:32:20.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:20.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj307 testdata.562265 2024-09-18T16:32:20.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:20.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj308 testdata.562265 2024-09-18T16:32:20.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:20.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj309 testdata.562265 2024-09-18T16:32:20.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:20.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj310 testdata.562265 2024-09-18T16:32:20.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:20.797 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj311 testdata.562265 2024-09-18T16:32:20.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:20.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj312 testdata.562265 2024-09-18T16:32:21.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:21.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj313 testdata.562265 2024-09-18T16:32:21.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:21.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj314 testdata.562265 2024-09-18T16:32:21.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:21.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj315 testdata.562265 2024-09-18T16:32:21.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:21.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj316 testdata.562265 2024-09-18T16:32:21.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:21.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj317 testdata.562265 2024-09-18T16:32:21.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:21.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj318 testdata.562265 2024-09-18T16:32:22.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:22.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj319 testdata.562265 2024-09-18T16:32:22.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:22.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj320 testdata.562265 2024-09-18T16:32:22.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:22.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj321 testdata.562265 2024-09-18T16:32:22.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:22.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj322 testdata.562265 2024-09-18T16:32:22.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:22.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj323 testdata.562265 2024-09-18T16:32:22.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:22.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj324 testdata.562265 2024-09-18T16:32:22.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:22.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj325 testdata.562265 2024-09-18T16:32:23.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:23.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj326 testdata.562265 2024-09-18T16:32:23.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:23.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj327 testdata.562265 2024-09-18T16:32:23.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:23.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj328 testdata.562265 2024-09-18T16:32:23.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:23.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj329 testdata.562265 2024-09-18T16:32:23.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:23.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj330 testdata.562265 2024-09-18T16:32:23.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:23.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj331 testdata.562265 2024-09-18T16:32:24.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:24.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj332 testdata.562265 2024-09-18T16:32:24.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:24.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj333 testdata.562265 2024-09-18T16:32:24.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:24.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj334 testdata.562265 2024-09-18T16:32:24.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:24.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj335 testdata.562265 2024-09-18T16:32:24.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:24.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj336 testdata.562265 2024-09-18T16:32:24.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:24.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj337 testdata.562265 2024-09-18T16:32:24.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:24.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj338 testdata.562265 2024-09-18T16:32:25.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:25.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj339 testdata.562265 2024-09-18T16:32:25.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:25.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj340 testdata.562265 2024-09-18T16:32:25.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:25.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj341 testdata.562265 2024-09-18T16:32:25.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:25.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj342 testdata.562265 2024-09-18T16:32:26.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:26.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj343 testdata.562265 2024-09-18T16:32:26.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:26.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj344 testdata.562265 2024-09-18T16:32:26.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:26.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj345 testdata.562265 2024-09-18T16:32:26.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:26.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj346 testdata.562265 2024-09-18T16:32:26.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:26.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj347 testdata.562265 2024-09-18T16:32:26.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:26.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj348 testdata.562265 2024-09-18T16:32:26.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:26.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj349 testdata.562265 2024-09-18T16:32:27.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:27.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj350 testdata.562265 2024-09-18T16:32:27.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:27.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj351 testdata.562265 2024-09-18T16:32:27.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:27.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj352 testdata.562265 2024-09-18T16:32:27.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:27.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj353 testdata.562265 2024-09-18T16:32:27.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:27.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj354 testdata.562265 2024-09-18T16:32:27.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:27.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj355 testdata.562265 2024-09-18T16:32:27.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:27.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj356 testdata.562265 2024-09-18T16:32:28.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:28.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj357 testdata.562265 2024-09-18T16:32:28.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:28.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj358 testdata.562265 2024-09-18T16:32:28.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:28.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj359 testdata.562265 2024-09-18T16:32:28.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:28.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj360 testdata.562265 2024-09-18T16:32:28.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:28.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj361 testdata.562265 2024-09-18T16:32:28.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:28.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj362 testdata.562265 2024-09-18T16:32:29.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:29.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj363 testdata.562265 2024-09-18T16:32:29.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:29.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj364 testdata.562265 2024-09-18T16:32:29.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:29.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj365 testdata.562265 2024-09-18T16:32:29.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:29.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj366 testdata.562265 2024-09-18T16:32:29.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:29.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj367 testdata.562265 2024-09-18T16:32:29.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:29.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj368 testdata.562265 2024-09-18T16:32:29.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:29.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj369 testdata.562265 2024-09-18T16:32:30.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:30.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj370 testdata.562265 2024-09-18T16:32:30.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:30.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj371 testdata.562265 2024-09-18T16:32:30.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:30.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj372 testdata.562265 2024-09-18T16:32:30.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:30.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj373 testdata.562265 2024-09-18T16:32:30.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:30.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj374 testdata.562265 2024-09-18T16:32:30.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:30.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj375 testdata.562265 2024-09-18T16:32:30.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:30.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj376 testdata.562265 2024-09-18T16:32:31.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:31.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj377 testdata.562265 2024-09-18T16:32:31.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:31.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj378 testdata.562265 2024-09-18T16:32:31.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:31.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj379 testdata.562265 2024-09-18T16:32:31.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:31.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj380 testdata.562265 2024-09-18T16:32:31.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:31.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj381 testdata.562265 2024-09-18T16:32:31.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:31.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj382 testdata.562265 2024-09-18T16:32:31.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:31.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj383 testdata.562265 2024-09-18T16:32:32.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:32.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj384 testdata.562265 2024-09-18T16:32:32.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:32.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj385 testdata.562265 2024-09-18T16:32:32.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:32.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj386 testdata.562265 2024-09-18T16:32:32.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:32.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj387 testdata.562265 2024-09-18T16:32:32.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:32.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj388 testdata.562265 2024-09-18T16:32:32.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:32.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj389 testdata.562265 2024-09-18T16:32:32.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:32.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj390 testdata.562265 2024-09-18T16:32:33.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:33.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj391 testdata.562265 2024-09-18T16:32:33.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:33.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj392 testdata.562265 2024-09-18T16:32:33.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:33.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj393 testdata.562265 2024-09-18T16:32:33.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:33.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj394 testdata.562265 2024-09-18T16:32:33.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:33.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj395 testdata.562265 2024-09-18T16:32:33.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:33.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj396 testdata.562265 2024-09-18T16:32:33.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:33.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj397 testdata.562265 2024-09-18T16:32:34.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:34.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj398 testdata.562265 2024-09-18T16:32:34.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:34.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj399 testdata.562265 2024-09-18T16:32:34.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:34.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj400 testdata.562265 2024-09-18T16:32:34.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:34.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj401 testdata.562265 2024-09-18T16:32:34.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:34.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj402 testdata.562265 2024-09-18T16:32:34.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:34.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj403 testdata.562265 2024-09-18T16:32:35.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:35.106 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj404 testdata.562265 2024-09-18T16:32:35.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:35.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj405 testdata.562265 2024-09-18T16:32:35.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:35.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj406 testdata.562265 2024-09-18T16:32:35.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:35.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj407 testdata.562265 2024-09-18T16:32:35.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:35.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj408 testdata.562265 2024-09-18T16:32:35.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:35.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj409 testdata.562265 2024-09-18T16:32:35.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:35.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj410 testdata.562265 2024-09-18T16:32:36.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:36.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj411 testdata.562265 2024-09-18T16:32:36.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:36.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj412 testdata.562265 2024-09-18T16:32:36.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:36.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj413 testdata.562265 2024-09-18T16:32:36.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:36.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj414 testdata.562265 2024-09-18T16:32:36.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:36.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj415 testdata.562265 2024-09-18T16:32:36.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:36.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj416 testdata.562265 2024-09-18T16:32:37.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:37.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj417 testdata.562265 2024-09-18T16:32:37.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:37.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj418 testdata.562265 2024-09-18T16:32:37.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:37.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj419 testdata.562265 2024-09-18T16:32:37.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:37.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj420 testdata.562265 2024-09-18T16:32:37.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:37.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj421 testdata.562265 2024-09-18T16:32:37.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:37.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj422 testdata.562265 2024-09-18T16:32:38.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:38.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj423 testdata.562265 2024-09-18T16:32:38.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:38.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj424 testdata.562265 2024-09-18T16:32:38.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:38.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj425 testdata.562265 2024-09-18T16:32:38.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:38.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj426 testdata.562265 2024-09-18T16:32:38.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:38.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj427 testdata.562265 2024-09-18T16:32:38.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:38.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj428 testdata.562265 2024-09-18T16:32:38.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:38.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj429 testdata.562265 2024-09-18T16:32:39.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:39.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj430 testdata.562265 2024-09-18T16:32:39.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:39.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj431 testdata.562265 2024-09-18T16:32:39.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:39.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj432 testdata.562265 2024-09-18T16:32:39.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:39.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj433 testdata.562265 2024-09-18T16:32:39.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:39.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj434 testdata.562265 2024-09-18T16:32:39.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:39.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj435 testdata.562265 2024-09-18T16:32:39.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:39.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj436 testdata.562265 2024-09-18T16:32:40.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:40.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj437 testdata.562265 2024-09-18T16:32:40.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:40.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj438 testdata.562265 2024-09-18T16:32:40.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:40.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj439 testdata.562265 2024-09-18T16:32:40.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:40.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj440 testdata.562265 2024-09-18T16:32:40.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:40.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj441 testdata.562265 2024-09-18T16:32:40.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:40.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj442 testdata.562265 2024-09-18T16:32:41.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:41.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj443 testdata.562265 2024-09-18T16:32:41.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:41.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj444 testdata.562265 2024-09-18T16:32:41.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:41.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj445 testdata.562265 2024-09-18T16:32:41.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:41.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj446 testdata.562265 2024-09-18T16:32:41.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:41.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj447 testdata.562265 2024-09-18T16:32:41.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:41.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj448 testdata.562265 2024-09-18T16:32:42.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:42.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj449 testdata.562265 2024-09-18T16:32:42.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:42.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj450 testdata.562265 2024-09-18T16:32:42.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:42.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj451 testdata.562265 2024-09-18T16:32:42.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:42.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj452 testdata.562265 2024-09-18T16:32:42.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:42.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj453 testdata.562265 2024-09-18T16:32:42.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:42.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj454 testdata.562265 2024-09-18T16:32:42.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:42.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj455 testdata.562265 2024-09-18T16:32:43.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:43.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj456 testdata.562265 2024-09-18T16:32:43.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:43.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj457 testdata.562265 2024-09-18T16:32:43.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:43.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj458 testdata.562265 2024-09-18T16:32:43.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:43.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj459 testdata.562265 2024-09-18T16:32:43.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:43.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj460 testdata.562265 2024-09-18T16:32:43.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:43.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj461 testdata.562265 2024-09-18T16:32:43.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:43.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj462 testdata.562265 2024-09-18T16:32:44.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:44.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj463 testdata.562265 2024-09-18T16:32:44.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:44.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj464 testdata.562265 2024-09-18T16:32:44.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:44.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj465 testdata.562265 2024-09-18T16:32:44.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:44.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj466 testdata.562265 2024-09-18T16:32:44.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:44.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj467 testdata.562265 2024-09-18T16:32:44.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:44.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj468 testdata.562265 2024-09-18T16:32:44.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:44.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj469 testdata.562265 2024-09-18T16:32:45.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:45.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj470 testdata.562265 2024-09-18T16:32:45.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:45.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj471 testdata.562265 2024-09-18T16:32:45.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:45.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj472 testdata.562265 2024-09-18T16:32:45.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:45.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj473 testdata.562265 2024-09-18T16:32:45.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:45.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj474 testdata.562265 2024-09-18T16:32:45.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:45.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj475 testdata.562265 2024-09-18T16:32:46.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:46.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj476 testdata.562265 2024-09-18T16:32:46.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:46.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj477 testdata.562265 2024-09-18T16:32:46.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:46.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj478 testdata.562265 2024-09-18T16:32:46.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:46.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj479 testdata.562265 2024-09-18T16:32:46.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:46.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj480 testdata.562265 2024-09-18T16:32:46.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:46.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj481 testdata.562265 2024-09-18T16:32:46.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:46.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj482 testdata.562265 2024-09-18T16:32:47.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:47.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj483 testdata.562265 2024-09-18T16:32:47.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:47.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj484 testdata.562265 2024-09-18T16:32:47.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:47.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj485 testdata.562265 2024-09-18T16:32:47.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:47.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj486 testdata.562265 2024-09-18T16:32:47.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:47.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj487 testdata.562265 2024-09-18T16:32:47.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:47.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj488 testdata.562265 2024-09-18T16:32:48.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:48.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj489 testdata.562265 2024-09-18T16:32:48.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:48.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj490 testdata.562265 2024-09-18T16:32:48.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:48.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj491 testdata.562265 2024-09-18T16:32:48.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:48.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj492 testdata.562265 2024-09-18T16:32:48.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:48.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj493 testdata.562265 2024-09-18T16:32:48.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:48.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj494 testdata.562265 2024-09-18T16:32:49.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:49.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj495 testdata.562265 2024-09-18T16:32:49.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:49.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj496 testdata.562265 2024-09-18T16:32:49.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:49.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj497 testdata.562265 2024-09-18T16:32:49.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:49.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj498 testdata.562265 2024-09-18T16:32:49.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:49.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj499 testdata.562265 2024-09-18T16:32:49.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:49.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj500 testdata.562265 2024-09-18T16:32:49.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:49.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj501 testdata.562265 2024-09-18T16:32:49.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:49.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj502 testdata.562265 2024-09-18T16:32:50.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:50.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj503 testdata.562265 2024-09-18T16:32:50.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:50.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj504 testdata.562265 2024-09-18T16:32:50.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:50.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj505 testdata.562265 2024-09-18T16:32:50.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:50.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj506 testdata.562265 2024-09-18T16:32:50.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:50.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj507 testdata.562265 2024-09-18T16:32:50.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:50.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj508 testdata.562265 2024-09-18T16:32:51.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:51.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj509 testdata.562265 2024-09-18T16:32:51.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:51.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj510 testdata.562265 2024-09-18T16:32:51.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:51.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj511 testdata.562265 2024-09-18T16:32:51.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:51.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj512 testdata.562265 2024-09-18T16:32:51.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:51.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj513 testdata.562265 2024-09-18T16:32:51.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:51.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj514 testdata.562265 2024-09-18T16:32:52.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:52.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj515 testdata.562265 2024-09-18T16:32:52.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:52.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj516 testdata.562265 2024-09-18T16:32:52.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:52.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj517 testdata.562265 2024-09-18T16:32:52.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:52.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj518 testdata.562265 2024-09-18T16:32:52.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:52.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj519 testdata.562265 2024-09-18T16:32:52.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:52.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj520 testdata.562265 2024-09-18T16:32:53.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:53.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj521 testdata.562265 2024-09-18T16:32:53.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:53.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj522 testdata.562265 2024-09-18T16:32:53.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:53.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj523 testdata.562265 2024-09-18T16:32:53.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:53.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj524 testdata.562265 2024-09-18T16:32:53.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:53.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj525 testdata.562265 2024-09-18T16:32:53.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:53.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj526 testdata.562265 2024-09-18T16:32:53.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:53.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj527 testdata.562265 2024-09-18T16:32:54.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:54.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj528 testdata.562265 2024-09-18T16:32:54.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:54.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj529 testdata.562265 2024-09-18T16:32:54.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:54.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj530 testdata.562265 2024-09-18T16:32:54.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:54.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj531 testdata.562265 2024-09-18T16:32:54.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:54.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj532 testdata.562265 2024-09-18T16:32:54.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:54.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj533 testdata.562265 2024-09-18T16:32:54.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:54.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj534 testdata.562265 2024-09-18T16:32:55.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:55.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj535 testdata.562265 2024-09-18T16:32:55.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:55.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj536 testdata.562265 2024-09-18T16:32:55.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:55.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj537 testdata.562265 2024-09-18T16:32:55.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:55.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj538 testdata.562265 2024-09-18T16:32:55.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:55.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj539 testdata.562265 2024-09-18T16:32:55.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:55.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj540 testdata.562265 2024-09-18T16:32:56.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:56.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj541 testdata.562265 2024-09-18T16:32:56.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:56.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj542 testdata.562265 2024-09-18T16:32:56.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:56.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj543 testdata.562265 2024-09-18T16:32:56.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:56.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj544 testdata.562265 2024-09-18T16:32:56.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:56.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj545 testdata.562265 2024-09-18T16:32:57.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:57.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj546 testdata.562265 2024-09-18T16:32:57.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:57.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj547 testdata.562265 2024-09-18T16:32:57.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:57.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj548 testdata.562265 2024-09-18T16:32:57.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:57.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj549 testdata.562265 2024-09-18T16:32:57.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:57.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj550 testdata.562265 2024-09-18T16:32:57.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:57.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj551 testdata.562265 2024-09-18T16:32:57.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:57.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj552 testdata.562265 2024-09-18T16:32:58.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:58.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj553 testdata.562265 2024-09-18T16:32:58.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:58.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj554 testdata.562265 2024-09-18T16:32:58.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:58.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj555 testdata.562265 2024-09-18T16:32:58.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:58.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj556 testdata.562265 2024-09-18T16:32:58.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:58.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj557 testdata.562265 2024-09-18T16:32:58.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:58.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj558 testdata.562265 2024-09-18T16:32:58.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:58.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj559 testdata.562265 2024-09-18T16:32:59.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:59.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj560 testdata.562265 2024-09-18T16:32:59.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:59.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj561 testdata.562265 2024-09-18T16:32:59.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:59.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj562 testdata.562265 2024-09-18T16:32:59.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:59.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj563 testdata.562265 2024-09-18T16:32:59.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:59.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj564 testdata.562265 2024-09-18T16:32:59.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:59.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj565 testdata.562265 2024-09-18T16:32:59.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:32:59.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj566 testdata.562265 2024-09-18T16:33:00.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:00.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj567 testdata.562265 2024-09-18T16:33:00.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:00.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj568 testdata.562265 2024-09-18T16:33:00.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:00.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj569 testdata.562265 2024-09-18T16:33:00.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:00.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj570 testdata.562265 2024-09-18T16:33:00.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:00.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj571 testdata.562265 2024-09-18T16:33:00.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:00.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj572 testdata.562265 2024-09-18T16:33:01.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:01.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj573 testdata.562265 2024-09-18T16:33:01.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:01.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj574 testdata.562265 2024-09-18T16:33:01.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:01.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj575 testdata.562265 2024-09-18T16:33:01.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:01.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj576 testdata.562265 2024-09-18T16:33:01.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:01.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj577 testdata.562265 2024-09-18T16:33:01.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:01.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj578 testdata.562265 2024-09-18T16:33:01.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:01.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj579 testdata.562265 2024-09-18T16:33:02.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:02.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj580 testdata.562265 2024-09-18T16:33:02.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:02.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj581 testdata.562265 2024-09-18T16:33:02.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:02.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj582 testdata.562265 2024-09-18T16:33:02.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:02.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj583 testdata.562265 2024-09-18T16:33:02.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:02.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj584 testdata.562265 2024-09-18T16:33:02.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:02.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj585 testdata.562265 2024-09-18T16:33:02.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:02.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj586 testdata.562265 2024-09-18T16:33:03.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:03.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj587 testdata.562265 2024-09-18T16:33:03.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:03.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj588 testdata.562265 2024-09-18T16:33:03.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:03.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj589 testdata.562265 2024-09-18T16:33:03.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:03.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj590 testdata.562265 2024-09-18T16:33:03.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:03.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj591 testdata.562265 2024-09-18T16:33:03.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:03.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj592 testdata.562265 2024-09-18T16:33:03.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:03.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj593 testdata.562265 2024-09-18T16:33:04.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:04.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj594 testdata.562265 2024-09-18T16:33:04.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:04.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj595 testdata.562265 2024-09-18T16:33:04.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:04.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj596 testdata.562265 2024-09-18T16:33:04.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:04.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj597 testdata.562265 2024-09-18T16:33:04.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:04.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj598 testdata.562265 2024-09-18T16:33:04.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:04.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj599 testdata.562265 2024-09-18T16:33:05.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:05.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj600 testdata.562265 2024-09-18T16:33:05.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:05.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj601 testdata.562265 2024-09-18T16:33:05.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:05.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj602 testdata.562265 2024-09-18T16:33:05.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:05.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj603 testdata.562265 2024-09-18T16:33:05.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:05.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj604 testdata.562265 2024-09-18T16:33:05.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:05.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj605 testdata.562265 2024-09-18T16:33:05.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:05.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj606 testdata.562265 2024-09-18T16:33:06.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:06.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj607 testdata.562265 2024-09-18T16:33:06.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:06.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj608 testdata.562265 2024-09-18T16:33:06.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:06.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj609 testdata.562265 2024-09-18T16:33:06.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:06.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj610 testdata.562265 2024-09-18T16:33:06.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:06.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj611 testdata.562265 2024-09-18T16:33:06.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:06.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj612 testdata.562265 2024-09-18T16:33:07.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:07.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj613 testdata.562265 2024-09-18T16:33:07.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:07.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj614 testdata.562265 2024-09-18T16:33:07.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:07.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj615 testdata.562265 2024-09-18T16:33:07.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:07.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj616 testdata.562265 2024-09-18T16:33:07.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:07.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj617 testdata.562265 2024-09-18T16:33:07.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:07.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj618 testdata.562265 2024-09-18T16:33:07.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:07.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj619 testdata.562265 2024-09-18T16:33:08.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:08.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj620 testdata.562265 2024-09-18T16:33:08.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:08.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj621 testdata.562265 2024-09-18T16:33:08.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:08.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj622 testdata.562265 2024-09-18T16:33:08.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:08.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj623 testdata.562265 2024-09-18T16:33:08.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:08.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj624 testdata.562265 2024-09-18T16:33:08.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:08.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj625 testdata.562265 2024-09-18T16:33:08.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:08.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj626 testdata.562265 2024-09-18T16:33:09.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:09.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj627 testdata.562265 2024-09-18T16:33:09.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:09.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj628 testdata.562265 2024-09-18T16:33:09.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:09.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj629 testdata.562265 2024-09-18T16:33:09.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:09.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj630 testdata.562265 2024-09-18T16:33:09.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:09.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj631 testdata.562265 2024-09-18T16:33:09.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:09.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj632 testdata.562265 2024-09-18T16:33:09.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:09.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj633 testdata.562265 2024-09-18T16:33:10.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:10.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj634 testdata.562265 2024-09-18T16:33:10.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:10.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj635 testdata.562265 2024-09-18T16:33:10.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:10.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj636 testdata.562265 2024-09-18T16:33:10.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:10.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj637 testdata.562265 2024-09-18T16:33:10.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:10.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj638 testdata.562265 2024-09-18T16:33:10.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:10.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj639 testdata.562265 2024-09-18T16:33:10.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:10.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj640 testdata.562265 2024-09-18T16:33:11.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:11.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj641 testdata.562265 2024-09-18T16:33:11.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:11.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj642 testdata.562265 2024-09-18T16:33:11.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:11.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj643 testdata.562265 2024-09-18T16:33:11.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:11.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj644 testdata.562265 2024-09-18T16:33:11.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:11.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj645 testdata.562265 2024-09-18T16:33:11.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:11.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj646 testdata.562265 2024-09-18T16:33:11.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:11.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj647 testdata.562265 2024-09-18T16:33:12.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:12.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj648 testdata.562265 2024-09-18T16:33:12.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:12.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj649 testdata.562265 2024-09-18T16:33:12.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:12.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj650 testdata.562265 2024-09-18T16:33:12.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:12.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj651 testdata.562265 2024-09-18T16:33:12.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:12.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj652 testdata.562265 2024-09-18T16:33:12.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:12.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj653 testdata.562265 2024-09-18T16:33:13.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:13.102 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj654 testdata.562265 2024-09-18T16:33:13.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:13.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj655 testdata.562265 2024-09-18T16:33:13.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:13.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj656 testdata.562265 2024-09-18T16:33:13.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:13.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj657 testdata.562265 2024-09-18T16:33:13.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:13.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj658 testdata.562265 2024-09-18T16:33:13.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:13.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj659 testdata.562265 2024-09-18T16:33:13.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:13.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj660 testdata.562265 2024-09-18T16:33:14.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:14.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj661 testdata.562265 2024-09-18T16:33:14.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:14.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj662 testdata.562265 2024-09-18T16:33:14.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:14.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj663 testdata.562265 2024-09-18T16:33:14.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:14.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj664 testdata.562265 2024-09-18T16:33:14.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:14.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj665 testdata.562265 2024-09-18T16:33:14.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:14.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj666 testdata.562265 2024-09-18T16:33:15.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:15.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj667 testdata.562265 2024-09-18T16:33:15.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:15.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj668 testdata.562265 2024-09-18T16:33:15.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:15.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj669 testdata.562265 2024-09-18T16:33:15.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:15.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj670 testdata.562265 2024-09-18T16:33:15.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:15.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj671 testdata.562265 2024-09-18T16:33:15.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:15.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj672 testdata.562265 2024-09-18T16:33:15.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:15.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj673 testdata.562265 2024-09-18T16:33:16.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:16.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj674 testdata.562265 2024-09-18T16:33:16.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:16.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj675 testdata.562265 2024-09-18T16:33:16.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:16.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj676 testdata.562265 2024-09-18T16:33:16.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:16.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj677 testdata.562265 2024-09-18T16:33:16.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:16.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj678 testdata.562265 2024-09-18T16:33:16.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:16.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj679 testdata.562265 2024-09-18T16:33:16.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:16.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj680 testdata.562265 2024-09-18T16:33:17.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:17.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj681 testdata.562265 2024-09-18T16:33:17.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:17.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj682 testdata.562265 2024-09-18T16:33:17.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:17.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj683 testdata.562265 2024-09-18T16:33:17.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:17.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj684 testdata.562265 2024-09-18T16:33:17.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:17.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj685 testdata.562265 2024-09-18T16:33:17.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:17.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj686 testdata.562265 2024-09-18T16:33:17.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:17.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj687 testdata.562265 2024-09-18T16:33:18.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:18.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj688 testdata.562265 2024-09-18T16:33:18.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:18.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj689 testdata.562265 2024-09-18T16:33:18.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:18.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj690 testdata.562265 2024-09-18T16:33:18.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:18.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj691 testdata.562265 2024-09-18T16:33:18.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:18.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj692 testdata.562265 2024-09-18T16:33:18.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:18.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj693 testdata.562265 2024-09-18T16:33:19.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:19.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj694 testdata.562265 2024-09-18T16:33:19.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:19.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj695 testdata.562265 2024-09-18T16:33:19.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:19.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj696 testdata.562265 2024-09-18T16:33:19.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:19.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj697 testdata.562265 2024-09-18T16:33:19.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:19.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj698 testdata.562265 2024-09-18T16:33:19.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:19.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj699 testdata.562265 2024-09-18T16:33:19.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:19.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj700 testdata.562265 2024-09-18T16:33:19.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:19.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj701 testdata.562265 2024-09-18T16:33:20.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:20.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj702 testdata.562265 2024-09-18T16:33:20.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:20.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj703 testdata.562265 2024-09-18T16:33:20.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:20.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj704 testdata.562265 2024-09-18T16:33:20.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:20.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj705 testdata.562265 2024-09-18T16:33:20.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:20.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj706 testdata.562265 2024-09-18T16:33:20.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:20.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj707 testdata.562265 2024-09-18T16:33:21.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:21.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj708 testdata.562265 2024-09-18T16:33:21.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:21.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj709 testdata.562265 2024-09-18T16:33:21.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:21.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj710 testdata.562265 2024-09-18T16:33:21.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:21.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj711 testdata.562265 2024-09-18T16:33:21.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:21.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj712 testdata.562265 2024-09-18T16:33:21.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:21.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj713 testdata.562265 2024-09-18T16:33:21.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:21.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj714 testdata.562265 2024-09-18T16:33:22.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:22.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj715 testdata.562265 2024-09-18T16:33:22.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:22.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj716 testdata.562265 2024-09-18T16:33:22.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:22.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj717 testdata.562265 2024-09-18T16:33:22.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:22.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj718 testdata.562265 2024-09-18T16:33:22.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:22.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj719 testdata.562265 2024-09-18T16:33:22.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:22.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj720 testdata.562265 2024-09-18T16:33:22.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:22.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj721 testdata.562265 2024-09-18T16:33:23.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:23.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj722 testdata.562265 2024-09-18T16:33:23.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:23.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj723 testdata.562265 2024-09-18T16:33:23.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:23.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj724 testdata.562265 2024-09-18T16:33:23.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:23.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj725 testdata.562265 2024-09-18T16:33:23.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:23.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj726 testdata.562265 2024-09-18T16:33:23.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:23.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj727 testdata.562265 2024-09-18T16:33:24.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:24.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj728 testdata.562265 2024-09-18T16:33:24.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:24.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj729 testdata.562265 2024-09-18T16:33:24.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:24.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj730 testdata.562265 2024-09-18T16:33:24.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:24.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj731 testdata.562265 2024-09-18T16:33:24.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:24.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj732 testdata.562265 2024-09-18T16:33:24.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:24.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj733 testdata.562265 2024-09-18T16:33:24.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:24.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj734 testdata.562265 2024-09-18T16:33:25.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:25.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj735 testdata.562265 2024-09-18T16:33:25.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:25.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj736 testdata.562265 2024-09-18T16:33:25.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:25.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj737 testdata.562265 2024-09-18T16:33:25.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:25.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj738 testdata.562265 2024-09-18T16:33:25.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:25.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj739 testdata.562265 2024-09-18T16:33:25.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:25.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj740 testdata.562265 2024-09-18T16:33:25.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:25.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj741 testdata.562265 2024-09-18T16:33:26.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:26.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj742 testdata.562265 2024-09-18T16:33:26.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:26.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj743 testdata.562265 2024-09-18T16:33:26.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:26.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj744 testdata.562265 2024-09-18T16:33:26.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:26.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj745 testdata.562265 2024-09-18T16:33:26.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:26.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj746 testdata.562265 2024-09-18T16:33:26.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:26.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj747 testdata.562265 2024-09-18T16:33:27.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:27.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj748 testdata.562265 2024-09-18T16:33:27.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:27.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj749 testdata.562265 2024-09-18T16:33:27.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:27.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj750 testdata.562265 2024-09-18T16:33:27.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:27.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj751 testdata.562265 2024-09-18T16:33:27.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:27.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj752 testdata.562265 2024-09-18T16:33:28.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:28.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj753 testdata.562265 2024-09-18T16:33:28.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:28.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj754 testdata.562265 2024-09-18T16:33:28.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:28.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj755 testdata.562265 2024-09-18T16:33:28.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:28.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj756 testdata.562265 2024-09-18T16:33:28.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:28.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj757 testdata.562265 2024-09-18T16:33:29.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:29.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj758 testdata.562265 2024-09-18T16:33:29.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:29.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj759 testdata.562265 2024-09-18T16:33:29.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:29.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj760 testdata.562265 2024-09-18T16:33:29.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:29.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj761 testdata.562265 2024-09-18T16:33:29.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:29.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj762 testdata.562265 2024-09-18T16:33:29.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:29.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj763 testdata.562265 2024-09-18T16:33:29.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:29.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj764 testdata.562265 2024-09-18T16:33:30.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:30.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj765 testdata.562265 2024-09-18T16:33:30.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:30.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj766 testdata.562265 2024-09-18T16:33:30.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:30.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj767 testdata.562265 2024-09-18T16:33:30.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:30.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj768 testdata.562265 2024-09-18T16:33:30.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:30.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj769 testdata.562265 2024-09-18T16:33:30.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:30.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj770 testdata.562265 2024-09-18T16:33:30.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:30.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj771 testdata.562265 2024-09-18T16:33:31.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:31.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj772 testdata.562265 2024-09-18T16:33:31.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:31.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj773 testdata.562265 2024-09-18T16:33:31.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:31.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj774 testdata.562265 2024-09-18T16:33:31.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:31.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj775 testdata.562265 2024-09-18T16:33:31.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:31.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj776 testdata.562265 2024-09-18T16:33:31.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:31.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj777 testdata.562265 2024-09-18T16:33:31.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:31.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj778 testdata.562265 2024-09-18T16:33:32.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:32.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj779 testdata.562265 2024-09-18T16:33:32.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:32.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj780 testdata.562265 2024-09-18T16:33:32.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:32.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj781 testdata.562265 2024-09-18T16:33:32.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:32.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj782 testdata.562265 2024-09-18T16:33:32.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:32.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj783 testdata.562265 2024-09-18T16:33:32.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:32.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj784 testdata.562265 2024-09-18T16:33:32.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:32.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj785 testdata.562265 2024-09-18T16:33:33.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:33.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj786 testdata.562265 2024-09-18T16:33:33.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:33.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj787 testdata.562265 2024-09-18T16:33:33.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:33.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj788 testdata.562265 2024-09-18T16:33:33.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:33.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj789 testdata.562265 2024-09-18T16:33:33.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:33.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj790 testdata.562265 2024-09-18T16:33:33.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:33.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj791 testdata.562265 2024-09-18T16:33:33.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:33.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj792 testdata.562265 2024-09-18T16:33:34.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:34.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj793 testdata.562265 2024-09-18T16:33:34.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:34.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj794 testdata.562265 2024-09-18T16:33:34.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:34.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj795 testdata.562265 2024-09-18T16:33:34.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:34.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj796 testdata.562265 2024-09-18T16:33:34.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:34.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj797 testdata.562265 2024-09-18T16:33:34.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:34.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj798 testdata.562265 2024-09-18T16:33:34.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:34.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj799 testdata.562265 2024-09-18T16:33:35.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:35.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj800 testdata.562265 2024-09-18T16:33:35.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:35.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj801 testdata.562265 2024-09-18T16:33:35.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:35.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj802 testdata.562265 2024-09-18T16:33:35.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:35.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj803 testdata.562265 2024-09-18T16:33:35.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:35.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj804 testdata.562265 2024-09-18T16:33:35.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:35.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj805 testdata.562265 2024-09-18T16:33:35.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:35.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj806 testdata.562265 2024-09-18T16:33:36.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:36.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj807 testdata.562265 2024-09-18T16:33:36.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:36.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj808 testdata.562265 2024-09-18T16:33:36.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:36.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj809 testdata.562265 2024-09-18T16:33:36.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:36.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj810 testdata.562265 2024-09-18T16:33:36.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:36.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj811 testdata.562265 2024-09-18T16:33:36.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:36.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj812 testdata.562265 2024-09-18T16:33:36.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:36.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj813 testdata.562265 2024-09-18T16:33:37.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:37.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj814 testdata.562265 2024-09-18T16:33:37.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:37.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj815 testdata.562265 2024-09-18T16:33:37.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:37.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj816 testdata.562265 2024-09-18T16:33:37.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:37.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj817 testdata.562265 2024-09-18T16:33:37.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:37.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj818 testdata.562265 2024-09-18T16:33:37.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:37.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj819 testdata.562265 2024-09-18T16:33:37.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:37.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj820 testdata.562265 2024-09-18T16:33:38.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:38.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj821 testdata.562265 2024-09-18T16:33:38.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:38.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj822 testdata.562265 2024-09-18T16:33:38.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:38.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj823 testdata.562265 2024-09-18T16:33:38.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:38.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj824 testdata.562265 2024-09-18T16:33:38.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:38.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj825 testdata.562265 2024-09-18T16:33:38.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:38.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj826 testdata.562265 2024-09-18T16:33:39.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:39.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj827 testdata.562265 2024-09-18T16:33:39.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:39.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj828 testdata.562265 2024-09-18T16:33:39.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:39.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj829 testdata.562265 2024-09-18T16:33:39.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:39.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj830 testdata.562265 2024-09-18T16:33:39.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:39.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj831 testdata.562265 2024-09-18T16:33:39.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:39.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj832 testdata.562265 2024-09-18T16:33:39.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:39.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj833 testdata.562265 2024-09-18T16:33:40.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:40.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj834 testdata.562265 2024-09-18T16:33:40.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:40.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj835 testdata.562265 2024-09-18T16:33:40.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:40.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj836 testdata.562265 2024-09-18T16:33:40.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:40.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj837 testdata.562265 2024-09-18T16:33:40.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:40.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj838 testdata.562265 2024-09-18T16:33:40.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:40.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj839 testdata.562265 2024-09-18T16:33:41.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:41.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj840 testdata.562265 2024-09-18T16:33:41.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:41.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj841 testdata.562265 2024-09-18T16:33:41.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:41.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj842 testdata.562265 2024-09-18T16:33:41.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:41.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj843 testdata.562265 2024-09-18T16:33:41.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:41.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj844 testdata.562265 2024-09-18T16:33:41.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:41.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj845 testdata.562265 2024-09-18T16:33:41.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:41.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj846 testdata.562265 2024-09-18T16:33:42.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:42.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj847 testdata.562265 2024-09-18T16:33:42.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:42.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj848 testdata.562265 2024-09-18T16:33:42.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:42.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj849 testdata.562265 2024-09-18T16:33:42.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:42.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj850 testdata.562265 2024-09-18T16:33:42.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:42.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj851 testdata.562265 2024-09-18T16:33:42.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:42.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj852 testdata.562265 2024-09-18T16:33:42.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:42.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj853 testdata.562265 2024-09-18T16:33:43.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:43.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj854 testdata.562265 2024-09-18T16:33:43.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:43.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj855 testdata.562265 2024-09-18T16:33:43.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:43.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj856 testdata.562265 2024-09-18T16:33:43.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:43.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj857 testdata.562265 2024-09-18T16:33:43.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:43.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj858 testdata.562265 2024-09-18T16:33:43.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:43.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj859 testdata.562265 2024-09-18T16:33:44.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:44.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj860 testdata.562265 2024-09-18T16:33:44.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:44.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj861 testdata.562265 2024-09-18T16:33:44.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:44.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj862 testdata.562265 2024-09-18T16:33:44.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:44.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj863 testdata.562265 2024-09-18T16:33:44.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:44.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj864 testdata.562265 2024-09-18T16:33:44.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:44.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj865 testdata.562265 2024-09-18T16:33:44.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:44.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj866 testdata.562265 2024-09-18T16:33:45.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:45.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj867 testdata.562265 2024-09-18T16:33:45.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:45.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj868 testdata.562265 2024-09-18T16:33:45.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:45.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj869 testdata.562265 2024-09-18T16:33:45.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:45.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj870 testdata.562265 2024-09-18T16:33:45.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:45.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj871 testdata.562265 2024-09-18T16:33:45.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:45.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj872 testdata.562265 2024-09-18T16:33:46.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:46.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj873 testdata.562265 2024-09-18T16:33:46.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:46.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj874 testdata.562265 2024-09-18T16:33:46.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:46.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj875 testdata.562265 2024-09-18T16:33:46.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:46.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj876 testdata.562265 2024-09-18T16:33:46.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:46.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj877 testdata.562265 2024-09-18T16:33:46.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:46.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj878 testdata.562265 2024-09-18T16:33:47.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:47.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj879 testdata.562265 2024-09-18T16:33:47.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:47.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj880 testdata.562265 2024-09-18T16:33:47.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:47.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj881 testdata.562265 2024-09-18T16:33:47.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:47.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj882 testdata.562265 2024-09-18T16:33:47.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:47.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj883 testdata.562265 2024-09-18T16:33:47.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:47.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj884 testdata.562265 2024-09-18T16:33:47.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:47.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj885 testdata.562265 2024-09-18T16:33:48.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:48.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj886 testdata.562265 2024-09-18T16:33:48.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:48.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj887 testdata.562265 2024-09-18T16:33:48.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:48.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj888 testdata.562265 2024-09-18T16:33:48.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:48.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj889 testdata.562265 2024-09-18T16:33:48.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:48.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj890 testdata.562265 2024-09-18T16:33:48.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:48.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj891 testdata.562265 2024-09-18T16:33:49.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:49.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj892 testdata.562265 2024-09-18T16:33:49.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:49.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj893 testdata.562265 2024-09-18T16:33:49.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:49.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj894 testdata.562265 2024-09-18T16:33:49.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:49.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj895 testdata.562265 2024-09-18T16:33:49.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:49.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj896 testdata.562265 2024-09-18T16:33:49.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:49.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj897 testdata.562265 2024-09-18T16:33:49.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:49.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj898 testdata.562265 2024-09-18T16:33:50.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:50.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj899 testdata.562265 2024-09-18T16:33:50.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:50.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj900 testdata.562265 2024-09-18T16:33:50.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:50.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj901 testdata.562265 2024-09-18T16:33:50.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:50.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj902 testdata.562265 2024-09-18T16:33:50.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:50.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj903 testdata.562265 2024-09-18T16:33:50.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:50.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj904 testdata.562265 2024-09-18T16:33:51.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:51.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj905 testdata.562265 2024-09-18T16:33:51.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:51.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj906 testdata.562265 2024-09-18T16:33:51.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:51.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj907 testdata.562265 2024-09-18T16:33:51.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:51.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj908 testdata.562265 2024-09-18T16:33:51.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:51.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj909 testdata.562265 2024-09-18T16:33:51.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:51.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj910 testdata.562265 2024-09-18T16:33:51.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:51.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj911 testdata.562265 2024-09-18T16:33:52.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:52.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj912 testdata.562265 2024-09-18T16:33:52.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:52.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj913 testdata.562265 2024-09-18T16:33:52.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:52.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj914 testdata.562265 2024-09-18T16:33:52.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:52.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj915 testdata.562265 2024-09-18T16:33:52.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:52.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj916 testdata.562265 2024-09-18T16:33:52.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:52.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj917 testdata.562265 2024-09-18T16:33:53.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:53.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj918 testdata.562265 2024-09-18T16:33:53.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:53.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj919 testdata.562265 2024-09-18T16:33:53.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:53.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj920 testdata.562265 2024-09-18T16:33:53.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:53.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj921 testdata.562265 2024-09-18T16:33:53.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:53.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj922 testdata.562265 2024-09-18T16:33:53.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:53.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj923 testdata.562265 2024-09-18T16:33:53.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:53.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj924 testdata.562265 2024-09-18T16:33:54.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:54.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj925 testdata.562265 2024-09-18T16:33:54.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:54.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj926 testdata.562265 2024-09-18T16:33:54.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:54.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj927 testdata.562265 2024-09-18T16:33:54.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:54.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj928 testdata.562265 2024-09-18T16:33:54.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:54.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj929 testdata.562265 2024-09-18T16:33:54.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:54.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj930 testdata.562265 2024-09-18T16:33:54.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:54.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj931 testdata.562265 2024-09-18T16:33:55.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:55.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj932 testdata.562265 2024-09-18T16:33:55.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:55.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj933 testdata.562265 2024-09-18T16:33:55.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:55.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj934 testdata.562265 2024-09-18T16:33:55.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:55.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj935 testdata.562265 2024-09-18T16:33:55.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:55.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj936 testdata.562265 2024-09-18T16:33:55.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:55.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj937 testdata.562265 2024-09-18T16:33:55.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:55.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj938 testdata.562265 2024-09-18T16:33:56.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:56.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj939 testdata.562265 2024-09-18T16:33:56.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:56.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj940 testdata.562265 2024-09-18T16:33:56.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:56.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj941 testdata.562265 2024-09-18T16:33:56.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:56.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj942 testdata.562265 2024-09-18T16:33:56.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:56.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj943 testdata.562265 2024-09-18T16:33:56.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:56.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj944 testdata.562265 2024-09-18T16:33:56.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:56.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj945 testdata.562265 2024-09-18T16:33:57.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:57.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj946 testdata.562265 2024-09-18T16:33:57.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:57.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj947 testdata.562265 2024-09-18T16:33:57.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:57.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj948 testdata.562265 2024-09-18T16:33:57.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:57.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj949 testdata.562265 2024-09-18T16:33:57.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:57.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj950 testdata.562265 2024-09-18T16:33:57.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:57.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj951 testdata.562265 2024-09-18T16:33:57.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:58.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj952 testdata.562265 2024-09-18T16:33:58.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:58.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj953 testdata.562265 2024-09-18T16:33:58.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:58.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj954 testdata.562265 2024-09-18T16:33:58.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:58.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj955 testdata.562265 2024-09-18T16:33:58.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:58.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj956 testdata.562265 2024-09-18T16:33:58.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:58.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj957 testdata.562265 2024-09-18T16:33:59.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:59.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj958 testdata.562265 2024-09-18T16:33:59.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:59.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj959 testdata.562265 2024-09-18T16:33:59.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:59.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj960 testdata.562265 2024-09-18T16:33:59.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:59.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj961 testdata.562265 2024-09-18T16:33:59.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:59.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj962 testdata.562265 2024-09-18T16:33:59.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:33:59.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj963 testdata.562265 2024-09-18T16:34:00.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:00.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj964 testdata.562265 2024-09-18T16:34:00.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:00.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj965 testdata.562265 2024-09-18T16:34:00.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:00.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj966 testdata.562265 2024-09-18T16:34:00.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:00.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj967 testdata.562265 2024-09-18T16:34:00.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:00.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj968 testdata.562265 2024-09-18T16:34:00.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:00.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj969 testdata.562265 2024-09-18T16:34:00.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:00.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj970 testdata.562265 2024-09-18T16:34:01.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:01.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj971 testdata.562265 2024-09-18T16:34:01.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:01.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj972 testdata.562265 2024-09-18T16:34:01.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:01.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj973 testdata.562265 2024-09-18T16:34:01.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:01.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj974 testdata.562265 2024-09-18T16:34:01.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:01.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj975 testdata.562265 2024-09-18T16:34:01.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:01.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj976 testdata.562265 2024-09-18T16:34:01.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:01.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj977 testdata.562265 2024-09-18T16:34:02.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:02.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj978 testdata.562265 2024-09-18T16:34:02.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:02.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj979 testdata.562265 2024-09-18T16:34:02.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:02.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj980 testdata.562265 2024-09-18T16:34:02.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:02.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj981 testdata.562265 2024-09-18T16:34:02.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:02.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj982 testdata.562265 2024-09-18T16:34:02.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:02.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj983 testdata.562265 2024-09-18T16:34:02.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:02.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj984 testdata.562265 2024-09-18T16:34:03.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:03.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj985 testdata.562265 2024-09-18T16:34:03.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:03.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj986 testdata.562265 2024-09-18T16:34:03.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:03.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj987 testdata.562265 2024-09-18T16:34:03.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:03.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj988 testdata.562265 2024-09-18T16:34:03.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:03.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj989 testdata.562265 2024-09-18T16:34:03.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:03.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj990 testdata.562265 2024-09-18T16:34:04.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:04.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj991 testdata.562265 2024-09-18T16:34:04.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:04.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj992 testdata.562265 2024-09-18T16:34:04.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:04.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj993 testdata.562265 2024-09-18T16:34:04.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:04.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj994 testdata.562265 2024-09-18T16:34:04.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:04.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj995 testdata.562265 2024-09-18T16:34:04.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:04.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj996 testdata.562265 2024-09-18T16:34:05.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:05.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj997 testdata.562265 2024-09-18T16:34:05.133 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:05.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj998 testdata.562265 2024-09-18T16:34:05.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:05.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj999 testdata.562265 2024-09-18T16:34:05.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: for i in `seq 1 $objects` 2024-09-18T16:34:05.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: rados -p test put obj1000 testdata.562265 2024-09-18T16:34:05.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:317: _scrub_abort: rm -f testdata.562265 2024-09-18T16:34:05.611 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:319: _scrub_abort: get_primary test obj1 2024-09-18T16:34:05.612 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T16:34:05.612 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T16:34:05.612 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T16:34:05.612 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T16:34:05.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:319: _scrub_abort: local primary=1 2024-09-18T16:34:05.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:320: _scrub_abort: local pgid=1.0 2024-09-18T16:34:05.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:322: _scrub_abort: ceph tell 1.0 scrub 2024-09-18T16:34:06.128 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:34:06.128 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T16:34:06.128 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T16:32:26.126916+0000" 2024-09-18T16:34:06.128 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:34:06.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:324: _scrub_abort: '[' scrub = deep_scrub ']' 2024-09-18T16:34:06.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: set -o pipefail 2024-09-18T16:34:06.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:331: _scrub_abort: found=no 2024-09-18T16:34:06.140 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: seq 0 200 2024-09-18T16:34:06.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: for i in $(seq 0 200) 2024-09-18T16:34:06.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: flush_pg_stats 2024-09-18T16:34:06.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=500 2024-09-18T16:34:06.143 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:34:06.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:34:06.444 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:34:06.444 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:34:06.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:34:06.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:34:06.444 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:34:06.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836519 2024-09-18T16:34:06.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836519 2024-09-18T16:34:06.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836519' 2024-09-18T16:34:06.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:34:06.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:34:06.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672997 2024-09-18T16:34:06.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672997 2024-09-18T16:34:06.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836519 1-42949672997' 2024-09-18T16:34:06.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:34:06.765 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:34:06.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509474 2024-09-18T16:34:06.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509474 2024-09-18T16:34:06.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836519 1-42949672997 2-64424509474' 2024-09-18T16:34:06.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:34:06.905 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836519 2024-09-18T16:34:06.906 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:34:06.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:34:06.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836519 2024-09-18T16:34:06.909 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:34:06.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836519 2024-09-18T16:34:06.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2024-09-18T16:34:06.912 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836519 2024-09-18T16:34:06.912 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:34:07.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836518 -lt 21474836519 2024-09-18T16:34:07.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:34:08.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-18T16:34:08.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:34:08.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836519 -lt 21474836519 2024-09-18T16:34:08.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:34:08.526 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672997 2024-09-18T16:34:08.526 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:34:08.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:34:08.528 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672997 2024-09-18T16:34:08.529 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:34:08.530 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672997 2024-09-18T16:34:08.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672997 2024-09-18T16:34:08.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672997' 2024-09-18T16:34:08.531 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:34:08.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672997 -lt 42949672997 2024-09-18T16:34:08.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:34:08.864 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509474 2024-09-18T16:34:08.864 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:34:08.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:34:08.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509474 2024-09-18T16:34:08.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:34:08.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509474 2024-09-18T16:34:08.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509474' 2024-09-18T16:34:08.870 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509474 2024-09-18T16:34:08.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:34:09.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509475 -lt 64424509474 2024-09-18T16:34:09.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: ceph pg dump pgs 2024-09-18T16:34:09.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: grep '^1.0' 2024-09-18T16:34:09.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: grep -q scrubbing 2024-09-18T16:34:09.472 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:34:09.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: for i in $(seq 0 200) 2024-09-18T16:34:09.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: flush_pg_stats 2024-09-18T16:34:09.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=500 2024-09-18T16:34:09.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:34:09.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:34:09.777 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:34:09.777 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:34:09.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:34:09.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:34:09.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:34:09.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836521 2024-09-18T16:34:09.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836521 2024-09-18T16:34:09.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521' 2024-09-18T16:34:09.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:34:09.912 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:34:10.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672998 2024-09-18T16:34:10.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672998 2024-09-18T16:34:10.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672998' 2024-09-18T16:34:10.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:34:10.044 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:34:10.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509476 2024-09-18T16:34:10.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509476 2024-09-18T16:34:10.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672998 2-64424509476' 2024-09-18T16:34:10.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:34:10.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836521 2024-09-18T16:34:10.183 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:34:10.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:34:10.185 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836521 2024-09-18T16:34:10.185 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:34:10.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836521 2024-09-18T16:34:10.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2024-09-18T16:34:10.187 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836521 2024-09-18T16:34:10.188 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:34:10.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836521 -lt 21474836521 2024-09-18T16:34:10.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:34:10.499 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672998 2024-09-18T16:34:10.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:34:10.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:34:10.503 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672998 2024-09-18T16:34:10.503 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:34:10.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672998 2024-09-18T16:34:10.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2024-09-18T16:34:10.504 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672998 2024-09-18T16:34:10.505 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:34:10.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672997 -lt 42949672998 2024-09-18T16:34:10.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:34:11.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-18T16:34:11.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:34:12.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672999 -lt 42949672998 2024-09-18T16:34:12.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:34:12.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509476 2024-09-18T16:34:12.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:34:12.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:34:12.158 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509476 2024-09-18T16:34:12.158 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:34:12.160 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509476 2024-09-18T16:34:12.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509476 2024-09-18T16:34:12.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509476' 2024-09-18T16:34:12.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:34:12.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509476 -lt 64424509476 2024-09-18T16:34:12.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: ceph pg dump pgs 2024-09-18T16:34:12.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: grep '^1.0' 2024-09-18T16:34:12.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: grep -q scrubbing 2024-09-18T16:34:12.830 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:34:12.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: for i in $(seq 0 200) 2024-09-18T16:34:12.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: flush_pg_stats 2024-09-18T16:34:12.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=500 2024-09-18T16:34:12.843 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:34:13.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:34:13.150 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:34:13.150 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:34:13.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:34:13.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:34:13.150 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:34:13.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836522 2024-09-18T16:34:13.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836522 2024-09-18T16:34:13.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836522' 2024-09-18T16:34:13.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:34:13.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:34:13.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673000 2024-09-18T16:34:13.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673000 2024-09-18T16:34:13.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836522 1-42949673000' 2024-09-18T16:34:13.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:34:13.487 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:34:13.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509478 2024-09-18T16:34:13.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509478 2024-09-18T16:34:13.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836522 1-42949673000 2-64424509478' 2024-09-18T16:34:13.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:34:13.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836522 2024-09-18T16:34:13.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:34:13.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:34:13.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836522 2024-09-18T16:34:13.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:34:13.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836522 2024-09-18T16:34:13.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836522' 2024-09-18T16:34:13.632 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836522 2024-09-18T16:34:13.633 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:34:13.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836521 -lt 21474836522 2024-09-18T16:34:13.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:34:14.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-18T16:34:14.975 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:34:15.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836522 -lt 21474836522 2024-09-18T16:34:15.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:34:15.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673000 2024-09-18T16:34:15.282 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:34:15.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:34:15.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673000 2024-09-18T16:34:15.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:34:15.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673000 2024-09-18T16:34:15.286 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949673000 2024-09-18T16:34:15.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673000' 2024-09-18T16:34:15.287 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:34:15.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673000 -lt 42949673000 2024-09-18T16:34:15.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:34:15.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509478 2024-09-18T16:34:15.644 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:34:15.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:34:15.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509478 2024-09-18T16:34:15.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:34:15.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509478 2024-09-18T16:34:15.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509478' 2024-09-18T16:34:15.649 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509478 2024-09-18T16:34:15.649 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:34:15.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509478 -lt 64424509478 2024-09-18T16:34:15.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: ceph pg dump pgs 2024-09-18T16:34:15.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: grep '^1.0' 2024-09-18T16:34:15.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: grep -q scrubbing 2024-09-18T16:34:16.304 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:34:16.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: found=yes 2024-09-18T16:34:16.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: break 2024-09-18T16:34:16.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:342: _scrub_abort: set +o pipefail 2024-09-18T16:34:16.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:344: _scrub_abort: test yes = no 2024-09-18T16:34:16.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:350: _scrub_abort: ceph osd set noscrub 2024-09-18T16:34:16.744 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is set 2024-09-18T16:34:16.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: '[' scrub = deep_scrub ']' 2024-09-18T16:34:16.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:357: _scrub_abort: set -o pipefail 2024-09-18T16:34:16.759 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:358: _scrub_abort: seq 0 200 2024-09-18T16:34:16.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:358: _scrub_abort: for i in $(seq 0 200) 2024-09-18T16:34:16.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:360: _scrub_abort: flush_pg_stats 2024-09-18T16:34:16.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=500 2024-09-18T16:34:16.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:34:17.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:34:17.136 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:34:17.136 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:34:17.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:34:17.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:34:17.137 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:34:17.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836524 2024-09-18T16:34:17.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836524 2024-09-18T16:34:17.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524' 2024-09-18T16:34:17.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:34:17.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:34:17.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673002 2024-09-18T16:34:17.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673002 2024-09-18T16:34:17.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949673002' 2024-09-18T16:34:17.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:34:17.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:34:17.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509479 2024-09-18T16:34:17.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509479 2024-09-18T16:34:17.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509479' 2024-09-18T16:34:17.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:34:17.601 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836524 2024-09-18T16:34:17.601 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:34:17.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:34:17.605 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836524 2024-09-18T16:34:17.605 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:34:17.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836524 2024-09-18T16:34:17.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-09-18T16:34:17.607 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836524 2024-09-18T16:34:17.607 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:34:17.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836523 -lt 21474836524 2024-09-18T16:34:17.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:34:18.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-18T16:34:18.919 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:34:19.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836524 -lt 21474836524 2024-09-18T16:34:19.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:34:19.226 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673002 2024-09-18T16:34:19.226 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:34:19.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:34:19.229 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673002 2024-09-18T16:34:19.229 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:34:19.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673002 2024-09-18T16:34:19.231 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949673002 2024-09-18T16:34:19.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673002' 2024-09-18T16:34:19.232 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:34:19.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673002 -lt 42949673002 2024-09-18T16:34:19.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:34:19.578 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509479 2024-09-18T16:34:19.578 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:34:19.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:34:19.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509479 2024-09-18T16:34:19.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:34:19.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509479 2024-09-18T16:34:19.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509479' 2024-09-18T16:34:19.583 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509479 2024-09-18T16:34:19.584 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:34:19.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509480 -lt 64424509479 2024-09-18T16:34:19.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:361: _scrub_abort: ceph pg dump pgs 2024-09-18T16:34:19.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:361: _scrub_abort: grep '^1.0' 2024-09-18T16:34:19.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:361: _scrub_abort: grep -q scrubbing 2024-09-18T16:34:20.291 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:34:20.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:363: _scrub_abort: continue 2024-09-18T16:34:20.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:358: _scrub_abort: for i in $(seq 0 200) 2024-09-18T16:34:20.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:360: _scrub_abort: flush_pg_stats 2024-09-18T16:34:20.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=500 2024-09-18T16:34:20.301 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:34:20.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:34:20.605 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:34:20.605 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:34:20.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:34:20.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:34:20.605 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:34:20.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836526 2024-09-18T16:34:20.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836526 2024-09-18T16:34:20.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836526' 2024-09-18T16:34:20.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:34:20.784 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:34:20.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673004 2024-09-18T16:34:20.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673004 2024-09-18T16:34:20.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836526 1-42949673004' 2024-09-18T16:34:20.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:34:20.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:34:21.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509481 2024-09-18T16:34:21.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509481 2024-09-18T16:34:21.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836526 1-42949673004 2-64424509481' 2024-09-18T16:34:21.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:34:21.069 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836526 2024-09-18T16:34:21.069 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:34:21.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:34:21.071 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836526 2024-09-18T16:34:21.071 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:34:21.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836526 2024-09-18T16:34:21.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2024-09-18T16:34:21.074 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836526 2024-09-18T16:34:21.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:34:21.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836525 -lt 21474836526 2024-09-18T16:34:21.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:34:22.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-18T16:34:22.377 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:34:22.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836526 -lt 21474836526 2024-09-18T16:34:22.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:34:22.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673004 2024-09-18T16:34:22.699 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:34:22.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:34:22.702 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:34:22.702 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673004 2024-09-18T16:34:22.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673004 2024-09-18T16:34:22.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673004' 2024-09-18T16:34:22.704 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949673004 2024-09-18T16:34:22.704 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:34:23.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673004 -lt 42949673004 2024-09-18T16:34:23.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:34:23.035 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509481 2024-09-18T16:34:23.035 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:34:23.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:34:23.038 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509481 2024-09-18T16:34:23.039 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:34:23.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509481 2024-09-18T16:34:23.041 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509481 2024-09-18T16:34:23.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509481' 2024-09-18T16:34:23.042 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:34:23.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509481 -lt 64424509481 2024-09-18T16:34:23.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:361: _scrub_abort: ceph pg dump pgs 2024-09-18T16:34:23.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:361: _scrub_abort: grep '^1.0' 2024-09-18T16:34:23.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:361: _scrub_abort: grep -q scrubbing 2024-09-18T16:34:23.716 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:34:23.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:366: _scrub_abort: break 2024-09-18T16:34:23.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:368: _scrub_abort: set +o pipefail 2024-09-18T16:34:23.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: sleep 5 2024-09-18T16:34:28.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: grep 'noscrub set, aborting' td/osd-scrub-test/osd.1.log 2024-09-18T16:34:28.752 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T16:34:17.909+0000 7f0d1b977700 10 osd.1 pg_epoch: 20 pg[1.0( v 19'1000 (0'0,19'1000] local-lis/les=17/18 n=1000 ec=17/17 lis/c=17/17 les/c/f=18/18/0 sis=17) [1,0,2] r=0 lpr=17 crt=19'1000 lcod 19'999 mlcod 19'999 active+clean+scrubbing [ 1.0: ] ] scrubber : noscrub set, aborting 2024-09-18T16:34:28.754 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: get_last_scrub_stamp 1.0 2024-09-18T16:34:28.754 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:34:28.754 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:34:28.754 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:34:28.755 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:34:29.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: local last_scrub=2024-09-11T16:32:26.126916+0000 2024-09-18T16:34:29.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: ceph config set osd osd_scrub_sleep 0.1 2024-09-18T16:34:29.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:381: _scrub_abort: ceph osd unset noscrub 2024-09-18T16:34:29.918 INFO:tasks.workunit.client.0.smithi046.stderr:noscrub is unset 2024-09-18T16:34:29.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: '[' scrub = deep_scrub ']' 2024-09-18T16:34:29.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:386: _scrub_abort: TIMEOUT=500 2024-09-18T16:34:29.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:387: _scrub_abort: wait_for_scrub 1.0 2024-09-11T16:32:26.126916+0000 2024-09-18T16:34:29.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-18T16:34:29.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-11T16:32:26.126916+0000 2024-09-18T16:34:29.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T16:34:29.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T16:34:29.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:34:29.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:34:29.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:34:29.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:34:29.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:34:29.934 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:34:30.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:34:30.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:34:31.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:34:31.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:34:31.293 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:34:31.293 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:34:31.293 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:34:31.294 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:34:31.294 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:34:31.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:34:31.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:34:32.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:34:32.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:34:32.596 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:34:32.596 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:34:32.596 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:34:32.596 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:34:32.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:34:32.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:34:32.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:34:33.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:34:33.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:34:33.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:34:33.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:34:33.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:34:33.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:34:33.912 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:34:34.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:34:34.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:34:35.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:34:35.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:34:35.221 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:34:35.221 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:34:35.221 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:34:35.224 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:34:35.224 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:34:35.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:34:35.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:34:36.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:34:36.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:34:36.534 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:34:36.534 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:34:36.534 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:34:36.535 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:34:36.535 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:34:36.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:34:36.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:34:37.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:34:37.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:34:37.849 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:34:37.849 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:34:37.849 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:34:37.849 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:34:37.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:34:38.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:34:38.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:34:39.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:34:39.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:34:39.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:34:39.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:34:39.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:34:39.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:34:39.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:34:39.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:34:39.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:34:40.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:34:40.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:34:40.463 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:34:40.463 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:34:40.463 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:34:40.464 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:34:40.464 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:34:40.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:34:40.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:34:41.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:34:41.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:34:41.774 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:34:41.775 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:34:41.775 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:34:41.775 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:34:41.775 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:34:42.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:34:42.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:34:43.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:34:43.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:34:43.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:34:43.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:34:43.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:34:43.076 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:34:43.076 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:34:43.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:34:43.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:34:44.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:34:44.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:34:44.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:34:44.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:34:44.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:34:44.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:34:44.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:34:44.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:34:44.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:34:45.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:34:45.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:34:45.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:34:45.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:34:45.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:34:45.699 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:34:45.699 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:34:46.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:34:46.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:34:47.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:34:47.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:34:47.008 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:34:47.008 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:34:47.008 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:34:47.009 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:34:47.009 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:34:47.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:34:47.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:34:48.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:34:48.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:34:48.318 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:34:48.318 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:34:48.318 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:34:48.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:34:48.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:34:48.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:34:48.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:34:49.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:34:49.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:34:49.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:34:49.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:34:49.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:34:49.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:34:49.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:34:49.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:34:49.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:34:50.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:34:50.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:34:50.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:34:50.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:34:50.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:34:50.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:34:50.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:34:51.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:34:51.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:34:52.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:34:52.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:34:52.240 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:34:52.240 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:34:52.240 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:34:52.240 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:34:52.241 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:34:52.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:34:52.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:34:53.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:34:53.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:34:53.551 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:34:53.551 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:34:53.551 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:34:53.552 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:34:53.552 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:34:53.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:34:53.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:34:54.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:34:54.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:34:54.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:34:54.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:34:54.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:34:54.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:34:54.870 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:34:55.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:34:55.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:34:56.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:34:56.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:34:56.228 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:34:56.228 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:34:56.228 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:34:56.228 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:34:56.229 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:34:56.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:34:56.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:34:57.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:34:57.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:34:57.534 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:34:57.534 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:34:57.534 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:34:57.535 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:34:57.535 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:34:57.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:34:57.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:34:58.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:34:58.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:34:58.844 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:34:58.844 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:34:58.844 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:34:58.845 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:34:58.845 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:34:59.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:34:59.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:00.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:00.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:00.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:00.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:00.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:00.156 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:00.156 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:00.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:00.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:01.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:01.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:01.469 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:01.469 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:01.469 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:01.470 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:01.470 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:01.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:01.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:02.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:02.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:02.780 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:02.780 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:02.780 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:02.780 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:02.780 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:03.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:03.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:04.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:04.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:04.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:04.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:04.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:04.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:04.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:04.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:04.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:05.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:05.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:05.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:05.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:05.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:05.399 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:05.399 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:05.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:05.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:06.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:06.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:06.702 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:06.702 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:06.702 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:06.702 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:06.702 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:07.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:07.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:08.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:08.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:08.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:08.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:08.072 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:08.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:08.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:08.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:08.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:09.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:09.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:09.378 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:09.378 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:09.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:09.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:09.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:09.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:09.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:10.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:10.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:10.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:10.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:10.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:10.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:10.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:11.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:11.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:12.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:12.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:12.045 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:12.046 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:12.046 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:12.046 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:12.046 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:12.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:12.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:13.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:13.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:13.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:13.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:13.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:13.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:13.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:13.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:13.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:14.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:14.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:14.666 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:14.666 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:14.666 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:14.667 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:14.667 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:14.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:14.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:15.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:15.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:15.975 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:15.975 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:15.975 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:15.976 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:15.976 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:16.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:16.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:17.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:17.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:17.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:17.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:17.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:17.313 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:17.313 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:17.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:17.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:18.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:18.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:18.619 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:18.619 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:18.619 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:18.620 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:18.620 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:18.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:18.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:19.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:19.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:19.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:19.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:19.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:19.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:19.934 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:20.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:20.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:21.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:21.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:21.243 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:21.243 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:21.243 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:21.244 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:21.244 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:21.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:21.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:22.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:22.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:22.553 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:22.553 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:22.553 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:22.553 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:22.554 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:22.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:22.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:23.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:23.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:23.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:23.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:23.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:23.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:23.866 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:24.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:24.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:25.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:25.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:25.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:25.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:25.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:25.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:25.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:25.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:25.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:26.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:26.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:26.492 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:26.492 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:26.492 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:26.492 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:26.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:26.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:26.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:27.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:27.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:27.801 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:27.801 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:27.801 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:27.802 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:27.802 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:28.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:28.111 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:29.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:29.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:29.114 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:29.114 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:29.114 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:29.114 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:29.114 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:29.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:29.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:30.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:30.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:30.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:30.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:30.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:30.426 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:30.426 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:30.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:30.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:31.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:31.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:31.737 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:31.737 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:31.737 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:31.738 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:31.738 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:32.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:32.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:33.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:33.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:33.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:33.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:33.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:33.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:33.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:33.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:33.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:34.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:34.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:34.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:34.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:34.374 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:34.374 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:34.374 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:34.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:34.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:35.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:35.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:35.680 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:35.680 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:35.680 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:35.680 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:35.681 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:35.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:35.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:36.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:36.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:36.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:36.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:36.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:36.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:36.997 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:37.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:37.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:38.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:38.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:38.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:38.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:38.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:38.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:38.310 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:38.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:38.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:39.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:39.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:39.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:39.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:39.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:39.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:39.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:39.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:39.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:40.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:40.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:40.926 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:40.926 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:40.926 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:40.927 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:40.927 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:41.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:41.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:42.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:42.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:42.238 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:42.238 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:42.238 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:42.239 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:42.239 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:42.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:42.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:43.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:43.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:43.558 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:43.558 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:43.559 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:43.559 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:43.559 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:43.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:43.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:44.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:44.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:44.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:44.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:44.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:44.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:44.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:45.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:45.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:46.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:46.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:46.184 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:46.185 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:46.185 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:46.185 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:46.185 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:46.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:46.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:47.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:47.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:47.488 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:47.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:47.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:47.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:47.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:47.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:47.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:48.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:48.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:48.803 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:48.803 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:48.803 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:48.803 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:48.803 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:49.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:49.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:50.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:50.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:50.115 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:50.115 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:50.115 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:50.116 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:50.116 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:50.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:50.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:51.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:51.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:51.426 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:51.426 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:51.426 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:51.427 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:51.427 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:51.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:51.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:52.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:52.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:52.735 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:52.735 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:52.735 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:52.735 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:52.736 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:53.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:53.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:54.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:54.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:54.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:54.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:54.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:54.076 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:54.076 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:54.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:54.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:55.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:55.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:55.388 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:55.388 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:55.388 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:55.389 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:55.389 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:55.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:55.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:56.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:56.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:56.694 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:56.694 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:56.694 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:56.694 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:56.695 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:57.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:57.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:58.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:58.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:58.007 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:58.007 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:58.008 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:58.008 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:58.009 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:58.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:58.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:35:59.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:35:59.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:35:59.318 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:35:59.318 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:35:59.318 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:35:59.318 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:35:59.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:35:59.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:35:59.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:00.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:00.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:00.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:00.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:00.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:00.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:00.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:00.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:00.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:01.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:01.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:01.926 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:01.926 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:01.926 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:01.927 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:01.927 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:02.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:02.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:03.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:03.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:03.244 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:03.244 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:03.244 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:03.245 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:03.245 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:03.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:03.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:04.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:04.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:04.555 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:04.555 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:04.555 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:04.556 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:04.556 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:04.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:04.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:05.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:05.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:05.872 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:05.872 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:05.872 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:05.873 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:05.873 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:06.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:06.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:07.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:07.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:07.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:07.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:07.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:07.192 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:07.192 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:07.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:07.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:08.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:08.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:08.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:08.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:08.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:08.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:08.501 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:08.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:08.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:09.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:09.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:09.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:09.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:09.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:09.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:09.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:10.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:10.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:11.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:11.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:11.125 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:11.125 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:11.125 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:11.125 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:11.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:11.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:11.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:12.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:12.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:12.441 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:12.441 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:12.441 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:12.441 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:12.441 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:12.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:12.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:13.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:13.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:13.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:13.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:13.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:13.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:13.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:14.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:14.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:15.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:15.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:15.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:15.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:15.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:15.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:15.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:15.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:15.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:16.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:16.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:16.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:16.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:16.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:16.374 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:16.374 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:16.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:16.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:17.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:17.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:17.680 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:17.680 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:17.680 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:17.680 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:17.681 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:17.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:17.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:18.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:18.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:18.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:18.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:18.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:18.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:18.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:19.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:19.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:20.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:20.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:20.292 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:20.292 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:20.292 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:20.292 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:20.293 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:20.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:20.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:21.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:21.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:21.602 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:21.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:21.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:21.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:21.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:21.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:21.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:22.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:22.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:22.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:22.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:22.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:22.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:22.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:23.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:23.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:24.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:24.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:24.228 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:24.228 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:24.228 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:24.229 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:24.229 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:24.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:24.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:25.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:25.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:25.538 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:25.538 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:25.538 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:25.538 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:25.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:25.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:25.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:26.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:26.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:26.847 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:26.847 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:26.848 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:26.848 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:26.848 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:27.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:27.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:28.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:28.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:28.159 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:28.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:28.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:28.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:28.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:28.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:28.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:29.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:29.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:29.467 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:29.467 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:29.467 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:29.467 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:29.468 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:29.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:29.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:30.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:30.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:30.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:30.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:30.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:30.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:30.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:31.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:31.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:32.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:32.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:32.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:32.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:32.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:32.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:32.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:32.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:32.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:33.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:33.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:33.391 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:33.391 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:33.391 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:33.391 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:33.392 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:33.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:33.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:34.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:34.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:34.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:34.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:34.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:34.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:34.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:35.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:35.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:36.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:36.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:36.006 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:36.006 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:36.006 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:36.006 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:36.007 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:36.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:36.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:37.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:37.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:37.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:37.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:37.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:37.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:37.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:37.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:37.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:38.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:38.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:38.635 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:38.635 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:38.635 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:38.635 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:38.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:38.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:38.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:39.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:39.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:39.949 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:39.949 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:39.949 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:39.950 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:39.950 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:40.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:40.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:41.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:41.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:41.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:41.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:41.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:41.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:41.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:41.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:41.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:42.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:42.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:42.565 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:42.565 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:42.565 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:42.566 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:42.566 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:42.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:42.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:43.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:43.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:43.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:43.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:43.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:43.886 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:43.886 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:44.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:44.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:45.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:45.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:45.189 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:45.190 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:45.190 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:45.190 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:45.190 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:45.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:45.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:46.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:46.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:46.501 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:46.501 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:46.501 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:46.502 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:46.502 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:46.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:46.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:47.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:47.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:47.819 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:47.819 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:47.819 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:47.820 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:47.820 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:48.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:48.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:49.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:49.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:49.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:49.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:49.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:49.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:49.127 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:49.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:49.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:50.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:50.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:50.438 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:50.438 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:50.438 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:50.438 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:50.439 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:50.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:50.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:51.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:51.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:51.746 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:51.746 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:51.746 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:51.746 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:51.747 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:52.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:52.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:53.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:53.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:53.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:53.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:53.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:53.071 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:53.071 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:53.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:53.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:54.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:54.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:54.374 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:54.374 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:54.374 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:54.375 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:54.375 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:54.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:54.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:55.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:55.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:55.687 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:55.687 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:55.687 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:55.688 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:55.688 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:55.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:55.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:56.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:56.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:56.995 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:56.995 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:56.995 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:56.995 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:56.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:57.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:57.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:58.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:58.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:58.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:58.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:58.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:58.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:58.305 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:58.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:58.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:36:59.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:36:59.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:36:59.611 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:36:59.611 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:36:59.611 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:36:59.612 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:36:59.612 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:36:59.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:36:59.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:00.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:00.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:00.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:00.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:00.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:00.916 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:00.916 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:01.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:01.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:02.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:02.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:02.232 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:02.232 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:02.232 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:02.233 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:02.233 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:02.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:02.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:03.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:03.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:03.536 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:03.536 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:03.536 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:03.536 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:03.537 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:03.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:03.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:04.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:04.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:04.839 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:04.840 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:04.840 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:04.840 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:04.840 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:05.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:05.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:06.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:06.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:06.156 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:06.156 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:06.156 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:06.156 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:06.156 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:06.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:06.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:07.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:07.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:07.458 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:07.458 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:07.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:07.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:07.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:07.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:07.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:08.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:08.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:08.772 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:08.772 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:08.772 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:08.772 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:08.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:09.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:09.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:10.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:10.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:10.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:10.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:10.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:10.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:10.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:10.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:10.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:11.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:11.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:11.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:11.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:11.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:11.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:11.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:11.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:11.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:12.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:12.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:12.703 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:12.703 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:12.703 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:12.704 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:12.704 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:13.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:13.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:14.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:14.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:14.016 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:14.016 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:14.016 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:14.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:14.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:14.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:14.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:15.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:15.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:15.339 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:15.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:15.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:15.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:15.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:15.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:15.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:16.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:16.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:16.659 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:16.659 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:16.659 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:16.659 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:16.660 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:16.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:16.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:17.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:17.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:17.971 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:17.971 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:17.971 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:17.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:17.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:18.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:18.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:19.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:19.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:19.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:19.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:19.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:19.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:19.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:19.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:19.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:20.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:20.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:20.590 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:20.590 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:20.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:20.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:20.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:20.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:20.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:21.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:21.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:21.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:21.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:21.902 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:21.902 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:21.902 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:22.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:22.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:23.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:23.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:23.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:23.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:23.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:23.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:23.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:23.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:23.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:24.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:24.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:24.525 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:24.525 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:24.525 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:24.525 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:24.526 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:24.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:24.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:25.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:25.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:25.835 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:25.835 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:25.835 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:25.836 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:25.836 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:26.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:26.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:27.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:27.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:27.142 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:27.142 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:27.142 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:27.142 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:27.143 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:27.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:27.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:28.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:28.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:28.455 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:28.455 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:28.455 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:28.456 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:28.456 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:28.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:28.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:29.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:29.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:29.763 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:29.763 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:29.763 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:29.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:29.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:30.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:30.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:31.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:31.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:31.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:31.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:31.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:31.095 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:31.095 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:31.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:31.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:32.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:32.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:32.399 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:32.399 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:32.399 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:32.399 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:32.399 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:32.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:32.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:33.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:33.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:33.710 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:33.710 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:33.710 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:33.710 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:33.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:34.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:34.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:35.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:35.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:35.021 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:35.021 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:35.021 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:35.021 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:35.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:35.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:35.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:36.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:36.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:36.328 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:36.328 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:36.328 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:36.329 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:36.329 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:36.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:36.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:37.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:37.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:37.640 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:37.640 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:37.640 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:37.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:37.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:37.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:37.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:38.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:38.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:38.947 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:38.947 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:38.947 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:38.947 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:38.948 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:39.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:39.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:40.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:40.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:40.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:40.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:40.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:40.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:40.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:40.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:40.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:41.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:41.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:41.559 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:41.559 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:41.559 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:41.560 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:41.560 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:41.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:41.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:42.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:42.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:42.866 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:42.866 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:42.866 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:42.866 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:42.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:43.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:43.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:44.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:44.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:44.170 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:44.170 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:44.170 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:44.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:44.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:44.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:44.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:45.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:45.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:45.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:45.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:45.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:45.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:45.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:45.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:45.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:46.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:46.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:46.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:46.793 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:46.793 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:46.793 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:46.793 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:47.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:47.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:48.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:48.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:48.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:48.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:48.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:48.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:48.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:48.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:48.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:49.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:49.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:49.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:49.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:49.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:49.406 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:49.406 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:49.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:49.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:50.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:50.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:50.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:50.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:50.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:50.714 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:50.714 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:51.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:51.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:52.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:52.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:52.033 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:52.033 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:52.034 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:52.034 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:52.034 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:52.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:52.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:53.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:53.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:53.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:53.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:53.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:53.346 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:53.347 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:53.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:53.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:54.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:54.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:54.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:54.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:54.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:54.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:54.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:54.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:54.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:55.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:55.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:55.954 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:55.954 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:55.954 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:55.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:55.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:56.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:56.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:57.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:57.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:57.264 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:57.264 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:57.264 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:57.264 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:57.265 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:57.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:57.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:58.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:58.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:58.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:58.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:58.573 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:58.574 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:58.574 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:37:58.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:37:58.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:37:59.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:37:59.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:37:59.882 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:37:59.882 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:37:59.882 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:37:59.882 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:37:59.882 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:38:00.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:38:00.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:38:01.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:38:01.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:38:01.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:38:01.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:38:01.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:38:01.192 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:38:01.192 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:38:01.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-11T16:32:26.126916+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:38:01.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:38:02.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:38:02.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:38:02.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:38:02.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:38:02.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:38:02.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:38:02.498 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:38:02.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:37:59.358968+0000 '>' 2024-09-11T16:32:26.126916+0000 2024-09-18T16:38:02.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T16:38:02.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: teardown td/osd-scrub-test 2024-09-18T16:38:02.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-09-18T16:38:02.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:38:02.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-18T16:38:02.807 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:38:02.807 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:38:02.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:38:02.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:38:02.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:38:02.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:38:02.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:38:02.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:38:02.926 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:38:02.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:38:02.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:38:02.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:38:02.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:38:02.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:38:02.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:38:02.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:38:02.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:38:02.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:38:02.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-09-18T16:38:02.979 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:38:02.979 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:38:02.979 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:38:02.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.562265 2024-09-18T16:38:02.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:38:02.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:38:02.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-09-18T16:38:02.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: setup td/osd-scrub-test 2024-09-18T16:38:02.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-test 2024-09-18T16:38:02.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-test 2024-09-18T16:38:02.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-09-18T16:38:02.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:38:02.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-18T16:38:02.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:38:02.983 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:38:02.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:38:02.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:38:02.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:38:02.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:38:02.987 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:38:02.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:38:02.988 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:38:02.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:38:02.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:38:02.991 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:38:02.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:38:02.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:38:02.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:38:02.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:38:02.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:38:02.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:38:02.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-09-18T16:38:02.998 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:38:02.999 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:38:02.999 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:38:02.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.562265 2024-09-18T16:38:03.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:38:03.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:38:03.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-test 2024-09-18T16:38:03.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T16:38:03.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:38:03.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:38:03.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.562265 2024-09-18T16:38:03.005 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T16:38:03.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T16:38:03.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T16:38:03.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-09-18T16:38:03.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: TEST_scrub_extended_sleep td/osd-scrub-test 2024-09-18T16:38:03.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:188: TEST_scrub_extended_sleep: local dir=td/osd-scrub-test 2024-09-18T16:38:03.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:189: TEST_scrub_extended_sleep: local poolname=test 2024-09-18T16:38:03.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:190: TEST_scrub_extended_sleep: local OSDS=3 2024-09-18T16:38:03.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:191: TEST_scrub_extended_sleep: local objects=15 2024-09-18T16:38:03.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:193: TEST_scrub_extended_sleep: TESTDATA=testdata.562265 2024-09-18T16:38:03.007 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:195: TEST_scrub_extended_sleep: date +%w 2024-09-18T16:38:03.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:195: TEST_scrub_extended_sleep: DAY=3 2024-09-18T16:38:03.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:197: TEST_scrub_extended_sleep: '[' 3 -ge 4 ']' 2024-09-18T16:38:03.008 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:202: TEST_scrub_extended_sleep: expr 3 + 2 2024-09-18T16:38:03.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:202: TEST_scrub_extended_sleep: DAY_START=5 2024-09-18T16:38:03.009 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:203: TEST_scrub_extended_sleep: expr 3 + 3 2024-09-18T16:38:03.011 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:203: TEST_scrub_extended_sleep: DAY_END=6 2024-09-18T16:38:03.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:205: TEST_scrub_extended_sleep: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2024-09-18T16:38:03.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-test 2024-09-18T16:38:03.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T16:38:03.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T16:38:03.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T16:38:03.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-test/a 2024-09-18T16:38:03.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2024-09-18T16:38:03.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T16:38:03.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:38:03.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:38:03.387 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:38:03.387 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:38:03.387 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:38:03.388 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:38:03.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-18T16:38:03.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T16:38:03.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T16:38:03.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:38:03.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:38:03.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T16:38:03.675 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T16:38:03.675 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:38:03.675 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:38:03.675 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:38:03.677 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:38:03.677 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:38:03.677 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:38:03.677 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.562265/ceph-mon.a.asok 2024-09-18T16:38:03.678 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:38:03.678 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.562265/ceph-mon.a.asok config get fsid 2024-09-18T16:38:03.782 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T16:38:03.782 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:38:03.782 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:38:03.782 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T16:38:03.782 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T16:38:03.783 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:38:03.783 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:38:03.783 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:38:03.783 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:38:03.783 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:38:03.783 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:38:03.784 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.562265/ceph-mon.a.asok 2024-09-18T16:38:03.784 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:38:03.784 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.562265/ceph-mon.a.asok config get mon_host 2024-09-18T16:38:03.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:206: TEST_scrub_extended_sleep: run_mgr td/osd-scrub-test x 2024-09-18T16:38:03.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-test 2024-09-18T16:38:03.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T16:38:03.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T16:38:03.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T16:38:03.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-test/x 2024-09-18T16:38:03.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T16:38:04.729 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T16:38:04.730 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:38:04.730 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:38:04.730 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:38:04.730 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:38:04.730 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:38:04.731 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:38:04.732 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:38:04.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:38:04.767 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:207: TEST_scrub_extended_sleep: expr 3 - 1 2024-09-18T16:38:04.769 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:207: TEST_scrub_extended_sleep: seq 0 2 2024-09-18T16:38:04.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:207: TEST_scrub_extended_sleep: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:38:04.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:209: TEST_scrub_extended_sleep: run_osd td/osd-scrub-test 0 --osd_scrub_sleep=0 --osd_scrub_extended_sleep=20 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_week_day=5 --osd_scrub_end_week_day=6 2024-09-18T16:38:04.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-09-18T16:38:04.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:38:04.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T16:38:04.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:38:04.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/0 2024-09-18T16:38:04.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-18T16:38:04.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:38:04.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:38:04.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:38:04.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-09-18T16:38:04.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-09-18T16:38:04.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:38:04.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:38:04.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-18T16:38:04.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:38:04.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:38:04.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:38:04.774 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:38:04.775 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:38:04.775 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:38:04.775 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:38:04.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:38:04.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:38:04.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:38:04.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:38:04.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-18T16:38:04.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-18T16:38:04.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:38:04.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:38:04.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:38:04.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:38:04.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:38:04.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_sleep=0 --osd_scrub_extended_sleep=20 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_week_day=5 --osd_scrub_end_week_day=6' 2024-09-18T16:38:04.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/0 2024-09-18T16:38:04.776 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:38:04.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=39e21f69-2d18-4725-be3b-81d1a8448ac3 2024-09-18T16:38:04.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 39e21f69-2d18-4725-be3b-81d1a8448ac3' 2024-09-18T16:38:04.777 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 39e21f69-2d18-4725-be3b-81d1a8448ac3 2024-09-18T16:38:04.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:38:04.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDsAetmdH4sLxAASPqZy+sYY37GLTLZY08Gcw== 2024-09-18T16:38:04.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDsAetmdH4sLxAASPqZy+sYY37GLTLZY08Gcw=="}' 2024-09-18T16:38:04.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 39e21f69-2d18-4725-be3b-81d1a8448ac3 -i td/osd-scrub-test/0/new.json 2024-09-18T16:38:05.089 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:38:05.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/0/new.json 2024-09-18T16:38:05.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_sleep=0 --osd_scrub_extended_sleep=20 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_week_day=5 --osd_scrub_end_week_day=6 --mkfs --key AQDsAetmdH4sLxAASPqZy+sYY37GLTLZY08Gcw== --osd-uuid 39e21f69-2d18-4725-be3b-81d1a8448ac3 2024-09-18T16:38:05.140 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:38:05.134+0000 7f504115d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:38:05.143 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:38:05.138+0000 7f504115d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:38:05.145 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:38:05.142+0000 7f504115d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:38:05.146 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:38:05.142+0000 7f504115d540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:38:05.146 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:38:05.142+0000 7f504115d540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:38:05.146 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:38:05.142+0000 7f504115d540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:38:05.226 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:38:05.222+0000 7f504115d540 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-09-18T16:38:11.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-09-18T16:38:11.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:38:11.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T16:38:11.517 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T16:38:11.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:38:11.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T16:38:11.888 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T16:38:11.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_sleep=0 --osd_scrub_extended_sleep=20 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_week_day=5 --osd_scrub_end_week_day=6 2024-09-18T16:38:11.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:38:11.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:38:11.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:38:11.921 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:38:11.918+0000 7fe4c5ec0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:38:11.923 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:38:11.918+0000 7fe4c5ec0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:38:11.925 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:38:11.922+0000 7fe4c5ec0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:38:12.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T16:38:12.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:38:12.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:38:12.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:38:12.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:38:12.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:12.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:38:12.255 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:38:12.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:12.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:38:12.459 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:38:12.454+0000 7fe4c5ec0540 -1 Falling back to public interface 2024-09-18T16:38:12.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:13.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:13.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:13.557 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:38:13.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:38:13.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:13.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:38:13.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:14.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:14.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:14.855 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:38:14.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:38:14.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:14.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:38:15.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:15.182 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:38:15.178+0000 7fe4c5ec0540 -1 osd.0 0 log_to_monitors true 2024-09-18T16:38:16.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:16.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:16.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:38:16.175 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:38:16.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:16.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:38:16.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:17.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:17.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:17.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:38:17.492 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:38:17.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:17.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:38:17.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:18.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:18.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:18.806 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:38:18.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:38:18.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:38:18.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:19.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:20.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:20.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:20.185 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:38:20.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:38:20.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:20.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:38:20.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:21.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:21.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:21.505 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:38:21.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:38:21.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:21.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:38:21.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:22.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:22.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:22.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:38:22.823 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:38:22.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:22.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:38:23.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:24.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:24.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:24.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:38:24.175 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:38:24.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:24.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:38:24.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:25.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:25.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:25.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:38:25.496 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:38:25.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:25.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:38:25.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:26.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:26.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:26.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:38:26.807 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:38:26.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:26.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:38:27.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:28.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:28.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:28.116 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:38:28.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:38:28.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:28.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:38:28.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:29.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:29.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:29.548 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:38:29.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:38:29.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:29.548 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:38:30.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:31.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:31.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:31.094 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:38:31.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:38:31.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:31.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:38:31.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:32.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:32.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:32.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:38:32.411 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:38:32.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:32.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:38:32.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:33.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:33.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:33.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:38:33.870 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:38:33.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:33.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:38:34.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:35.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:35.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:35.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:38:35.581 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:38:35.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:35.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:38:36.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:37.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:37.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:37.234 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:38:37.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:38:37.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:37.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:38:37.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:38.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:38.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:38.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T16:38:38.652 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T16:38:38.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:38.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:38:38.956 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/648386,v1:127.0.0.1:6803/648386] [v2:127.0.0.1:6804/648386,v1:127.0.0.1:6805/648386] exists,up 39e21f69-2d18-4725-be3b-81d1a8448ac3 2024-09-18T16:38:38.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:38:38.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:38:38.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:38:38.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:207: TEST_scrub_extended_sleep: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:38:38.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:209: TEST_scrub_extended_sleep: run_osd td/osd-scrub-test 1 --osd_scrub_sleep=0 --osd_scrub_extended_sleep=20 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_week_day=5 --osd_scrub_end_week_day=6 2024-09-18T16:38:38.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-09-18T16:38:38.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:38:38.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T16:38:38.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:38:38.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/1 2024-09-18T16:38:38.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-18T16:38:38.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:38:38.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:38:38.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:38:38.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-09-18T16:38:38.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-09-18T16:38:38.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:38:38.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:38:38.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-18T16:38:38.960 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:38:38.960 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:38:38.960 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:38:38.962 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:38:38.962 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:38:38.962 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:38:38.962 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:38:38.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:38:38.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:38:38.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:38:38.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:38:38.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-18T16:38:38.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-18T16:38:38.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:38:38.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:38:38.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:38:38.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:38:38.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:38:38.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_sleep=0 --osd_scrub_extended_sleep=20 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_week_day=5 --osd_scrub_end_week_day=6' 2024-09-18T16:38:38.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/1 2024-09-18T16:38:38.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:38:38.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=14cfdcbb-dada-4070-9c52-dbada4675877 2024-09-18T16:38:38.964 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 14cfdcbb-dada-4070-9c52-dbada4675877 2024-09-18T16:38:38.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 14cfdcbb-dada-4070-9c52-dbada4675877' 2024-09-18T16:38:38.965 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:38:38.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAOAutmIB65OhAANj8k+XeaSuukAQRKB4JKgg== 2024-09-18T16:38:38.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAOAutmIB65OhAANj8k+XeaSuukAQRKB4JKgg=="}' 2024-09-18T16:38:38.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 14cfdcbb-dada-4070-9c52-dbada4675877 -i td/osd-scrub-test/1/new.json 2024-09-18T16:38:39.367 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:38:39.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/1/new.json 2024-09-18T16:38:39.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_sleep=0 --osd_scrub_extended_sleep=20 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_week_day=5 --osd_scrub_end_week_day=6 --mkfs --key AQAOAutmIB65OhAANj8k+XeaSuukAQRKB4JKgg== --osd-uuid 14cfdcbb-dada-4070-9c52-dbada4675877 2024-09-18T16:38:39.422 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:38:39.417+0000 7fe142645540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:38:39.426 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:38:39.421+0000 7fe142645540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:38:39.428 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:38:39.425+0000 7fe142645540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:38:39.430 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:38:39.425+0000 7fe142645540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:38:39.430 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:38:39.425+0000 7fe142645540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:38:39.430 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:38:39.425+0000 7fe142645540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:38:39.509 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:38:39.505+0000 7fe142645540 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-09-18T16:38:45.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-09-18T16:38:45.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:38:45.822 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T16:38:45.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T16:38:45.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:38:46.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T16:38:46.205 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T16:38:46.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_sleep=0 --osd_scrub_extended_sleep=20 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_week_day=5 --osd_scrub_end_week_day=6 2024-09-18T16:38:46.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:38:46.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:38:46.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:38:46.238 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:38:46.233+0000 7f12b6724540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:38:46.240 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:38:46.233+0000 7f12b6724540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:38:46.245 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:38:46.237+0000 7f12b6724540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:38:46.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T16:38:46.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:38:46.554 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:38:46.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T16:38:46.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:38:46.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:38:46.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:46.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:38:46.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:46.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:38:46.783 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:38:46.777+0000 7f12b6724540 -1 Falling back to public interface 2024-09-18T16:38:46.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:47.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:47.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:47.948 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:38:47.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:38:47.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:47.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:38:48.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:49.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:49.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:49.263 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:38:49.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:38:49.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:49.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:38:49.571 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:38:49.565+0000 7f12b6724540 -1 osd.1 0 log_to_monitors true 2024-09-18T16:38:49.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:50.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:50.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:50.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:38:50.582 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:38:50.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:38:50.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:50.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:51.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:51.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:51.965 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:38:51.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:38:51.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:51.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:38:52.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:53.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:53.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:53.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:38:53.332 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:38:53.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:53.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:38:53.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:54.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:54.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:54.641 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:38:54.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:38:54.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:54.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:38:54.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:55.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:55.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:55.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:38:55.955 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:38:55.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:55.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:38:56.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:57.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:57.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:57.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:38:57.307 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:38:57.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:57.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:38:57.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:58.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:58.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:58.618 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:38:58.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:38:58.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:58.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:38:58.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:38:59.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:38:59.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:38:59.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:38:59.941 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:38:59.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:38:59.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:39:00.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:01.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:01.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:01.332 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:39:01.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:39:01.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:01.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:39:01.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:02.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:02.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:02.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:39:02.652 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:39:02.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:02.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:39:03.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:04.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:04.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:04.266 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:39:04.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:39:04.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:04.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:39:04.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:05.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:05.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:05.682 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:39:05.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:39:05.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:05.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:39:05.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:06.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:06.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:06.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:39:06.989 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:39:06.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:06.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:39:07.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:08.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:08.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:08.696 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:39:08.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:39:08.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:08.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:39:09.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:10.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:10.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:10.126 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:39:10.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:39:10.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:10.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:39:10.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:11.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:11.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:11.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:39:11.650 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:39:11.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:11.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:39:12.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:13.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:13.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:13.025 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T16:39:13.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T16:39:13.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:13.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:39:13.332 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/650252,v1:127.0.0.1:6811/650252] [v2:127.0.0.1:6812/650252,v1:127.0.0.1:6813/650252] exists,up 14cfdcbb-dada-4070-9c52-dbada4675877 2024-09-18T16:39:13.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:39:13.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:39:13.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:39:13.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:207: TEST_scrub_extended_sleep: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:39:13.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:209: TEST_scrub_extended_sleep: run_osd td/osd-scrub-test 2 --osd_scrub_sleep=0 --osd_scrub_extended_sleep=20 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_week_day=5 --osd_scrub_end_week_day=6 2024-09-18T16:39:13.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-09-18T16:39:13.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:39:13.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T16:39:13.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:39:13.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/2 2024-09-18T16:39:13.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-18T16:39:13.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:39:13.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:39:13.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:39:13.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2024-09-18T16:39:13.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2024-09-18T16:39:13.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:39:13.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:39:13.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-18T16:39:13.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:39:13.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:39:13.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:39:13.337 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:39:13.337 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:39:13.338 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:39:13.338 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:39:13.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:39:13.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:39:13.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:39:13.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:39:13.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-18T16:39:13.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-18T16:39:13.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:39:13.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:39:13.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:39:13.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:39:13.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:39:13.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_sleep=0 --osd_scrub_extended_sleep=20 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_week_day=5 --osd_scrub_end_week_day=6' 2024-09-18T16:39:13.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/2 2024-09-18T16:39:13.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:39:13.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3ce98f7b-6e44-4389-bf6e-644c851f6dad 2024-09-18T16:39:13.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 3ce98f7b-6e44-4389-bf6e-644c851f6dad' 2024-09-18T16:39:13.341 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 3ce98f7b-6e44-4389-bf6e-644c851f6dad 2024-09-18T16:39:13.341 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:39:13.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAxAutmfEJtFRAAk4LvzNa3AylMf/wt+l1ivw== 2024-09-18T16:39:13.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAxAutmfEJtFRAAk4LvzNa3AylMf/wt+l1ivw=="}' 2024-09-18T16:39:13.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3ce98f7b-6e44-4389-bf6e-644c851f6dad -i td/osd-scrub-test/2/new.json 2024-09-18T16:39:13.781 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:39:13.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/2/new.json 2024-09-18T16:39:13.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_sleep=0 --osd_scrub_extended_sleep=20 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_week_day=5 --osd_scrub_end_week_day=6 --mkfs --key AQAxAutmfEJtFRAAk4LvzNa3AylMf/wt+l1ivw== --osd-uuid 3ce98f7b-6e44-4389-bf6e-644c851f6dad 2024-09-18T16:39:13.831 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:39:13.825+0000 7f6f65f98540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:39:13.839 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:39:13.833+0000 7f6f65f98540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:39:13.841 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:39:13.837+0000 7f6f65f98540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:39:13.842 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:39:13.837+0000 7f6f65f98540 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-09-18T16:39:13.842 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:39:13.837+0000 7f6f65f98540 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-09-18T16:39:13.842 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:39:13.837+0000 7f6f65f98540 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-09-18T16:39:13.917 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:39:13.913+0000 7f6f65f98540 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-09-18T16:39:20.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2024-09-18T16:39:20.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:39:20.203 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T16:39:20.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T16:39:20.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:39:20.582 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T16:39:20.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T16:39:20.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_sleep=0 --osd_scrub_extended_sleep=20 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_week_day=5 --osd_scrub_end_week_day=6 2024-09-18T16:39:20.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:39:20.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:39:20.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:39:20.614 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:39:20.609+0000 7f32d872f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:39:20.617 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:39:20.613+0000 7f32d872f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:39:20.618 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:39:20.613+0000 7f32d872f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:39:20.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T16:39:20.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:39:20.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T16:39:20.945 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:39:20.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:39:20.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:39:20.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:20.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:39:20.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:39:20.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:21.163 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:39:21.157+0000 7f32d872f540 -1 Falling back to public interface 2024-09-18T16:39:21.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:22.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:22.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:22.314 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:39:22.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:39:22.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:22.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:39:22.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:23.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:23.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:23.625 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:39:23.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:39:23.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:23.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:39:23.876 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:39:23.869+0000 7f32d872f540 -1 osd.2 0 log_to_monitors true 2024-09-18T16:39:23.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:24.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:24.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:24.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:39:24.933 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:39:24.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:39:24.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:25.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:26.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:26.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:26.383 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:39:26.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:39:26.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:26.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:39:26.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:27.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:27.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:27.693 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:39:27.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:39:27.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:39:27.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:28.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:29.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:29.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:29.002 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:39:29.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:39:29.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:29.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:39:29.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:30.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:30.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:30.342 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:39:30.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:39:30.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:30.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:39:30.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:31.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:31.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:31.684 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:39:31.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:39:31.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:31.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:39:31.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:32.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:32.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:32.997 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:39:32.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:39:32.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:32.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:39:33.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:34.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:34.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:34.306 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:39:34.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:39:34.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:34.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:39:34.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:35.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:35.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:35.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:39:35.689 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:39:35.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:35.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:39:35.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:36.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:36.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:36.998 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:39:36.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:39:36.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:36.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:39:37.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:38.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:38.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:38.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:39:38.634 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:39:38.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:38.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:39:39.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:40.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:40.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:40.250 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:39:40.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:39:40.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:40.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:39:40.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:41.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:41.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:41.996 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:39:41.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:39:41.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:41.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:39:42.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:43.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:43.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:43.651 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:39:43.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:39:43.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:43.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:39:44.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:45.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:45.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:45.042 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:39:45.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:39:45.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:45.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:39:45.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:39:46.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:39:46.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:39:46.776 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:39:46.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:39:46.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:39:46.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:39:47.098 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/652056,v1:127.0.0.1:6819/652056] [v2:127.0.0.1:6820/652056,v1:127.0.0.1:6821/652056] exists,up 3ce98f7b-6e44-4389-bf6e-644c851f6dad 2024-09-18T16:39:47.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:39:47.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:39:47.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:39:47.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:220: TEST_scrub_extended_sleep: create_pool test 1 1 2024-09-18T16:39:47.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-18T16:39:47.573 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'test' created 2024-09-18T16:39:47.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T16:39:48.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:221: TEST_scrub_extended_sleep: wait_for_clean 2024-09-18T16:39:48.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:39:48.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:39:48.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:39:48.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:39:48.590 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:39:48.591 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:39:48.591 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:39:48.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:39:48.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:39:48.592 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:39:48.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:39:48.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:39:48.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:39:48.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=500 2024-09-18T16:39:48.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:39:49.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:39:49.222 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:39:49.222 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:39:49.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:39:49.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:39:49.223 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:39:49.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836501 2024-09-18T16:39:49.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836501 2024-09-18T16:39:49.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836501' 2024-09-18T16:39:49.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:39:49.364 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:39:49.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672976 2024-09-18T16:39:49.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672976 2024-09-18T16:39:49.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836501 1-42949672976' 2024-09-18T16:39:49.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:39:49.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:39:49.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509445 2024-09-18T16:39:49.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509445 2024-09-18T16:39:49.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836501 1-42949672976 2-64424509445' 2024-09-18T16:39:49.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:39:49.683 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836501 2024-09-18T16:39:49.683 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:39:49.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:39:49.686 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836501 2024-09-18T16:39:49.687 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:39:49.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836501 2024-09-18T16:39:49.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-18T16:39:49.689 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836501 2024-09-18T16:39:49.690 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:39:50.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836499 -lt 21474836501 2024-09-18T16:39:50.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:39:51.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-18T16:39:51.009 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:39:51.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836501 -lt 21474836501 2024-09-18T16:39:51.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:39:51.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672976 2024-09-18T16:39:51.320 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:39:51.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:39:51.323 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672976 2024-09-18T16:39:51.323 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:39:51.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672976 2024-09-18T16:39:51.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-18T16:39:51.326 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672976 2024-09-18T16:39:51.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:39:51.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672976 -lt 42949672976 2024-09-18T16:39:51.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:39:51.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509445 2024-09-18T16:39:51.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:39:51.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:39:51.661 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509445 2024-09-18T16:39:51.661 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:39:51.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509445 2024-09-18T16:39:51.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-09-18T16:39:51.663 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509445 2024-09-18T16:39:51.664 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:39:52.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509445 -lt 64424509445 2024-09-18T16:39:52.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:39:52.019 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:39:52.019 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:39:52.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:39:52.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:39:52.507 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:39:52.507 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:39:52.507 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:39:52.507 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:39:52.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:39:52.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:39:52.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:39:52.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:39:52.857 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:39:52.858 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:39:53.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:39:53.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:39:53.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:39:53.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:224: TEST_scrub_extended_sleep: get_pg test SOMETHING 2024-09-18T16:39:53.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-18T16:39:53.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T16:39:53.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test SOMETHING 2024-09-18T16:39:53.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T16:39:53.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:224: TEST_scrub_extended_sleep: local pgid=1.0 2024-09-18T16:39:53.561 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:225: TEST_scrub_extended_sleep: get_primary test SOMETHING 2024-09-18T16:39:53.561 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T16:39:53.561 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T16:39:53.561 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test SOMETHING 2024-09-18T16:39:53.562 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T16:39:53.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:225: TEST_scrub_extended_sleep: local primary=1 2024-09-18T16:39:53.887 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:226: TEST_scrub_extended_sleep: get_last_scrub_stamp 1.0 2024-09-18T16:39:53.887 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:39:53.887 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:39:53.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:39:53.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:39:54.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:226: TEST_scrub_extended_sleep: local last_scrub=2024-09-18T16:39:47.481174+0000 2024-09-18T16:39:54.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:227: TEST_scrub_extended_sleep: ceph tell 1.0 scrub 2024-09-18T16:39:54.302 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:39:54.302 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T16:39:54.302 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-11T16:38:14.301057+0000" 2024-09-18T16:39:54.302 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:39:54.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:230: TEST_scrub_extended_sleep: PASSED=false 2024-09-18T16:39:54.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:231: TEST_scrub_extended_sleep: (( i=0 )) 2024-09-18T16:39:54.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:231: TEST_scrub_extended_sleep: (( i < 15 )) 2024-09-18T16:39:54.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:232: TEST_scrub_extended_sleep: grep -q 'scrub state.*, sleeping' td/osd-scrub-test/osd.1.log 2024-09-18T16:39:54.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:237: TEST_scrub_extended_sleep: sleep 1 2024-09-18T16:39:55.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:231: TEST_scrub_extended_sleep: (( i++ )) 2024-09-18T16:39:55.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:231: TEST_scrub_extended_sleep: (( i < 15 )) 2024-09-18T16:39:55.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:232: TEST_scrub_extended_sleep: grep -q 'scrub state.*, sleeping' td/osd-scrub-test/osd.1.log 2024-09-18T16:39:55.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:237: TEST_scrub_extended_sleep: sleep 1 2024-09-18T16:39:56.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:231: TEST_scrub_extended_sleep: (( i++ )) 2024-09-18T16:39:56.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:231: TEST_scrub_extended_sleep: (( i < 15 )) 2024-09-18T16:39:56.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:232: TEST_scrub_extended_sleep: grep -q 'scrub state.*, sleeping' td/osd-scrub-test/osd.1.log 2024-09-18T16:39:56.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:234: TEST_scrub_extended_sleep: PASSED=true 2024-09-18T16:39:56.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:235: TEST_scrub_extended_sleep: break 2024-09-18T16:39:56.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:241: TEST_scrub_extended_sleep: '[' true = false ']' 2024-09-18T16:39:56.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:247: TEST_scrub_extended_sleep: ceph tell osd.1 config set osd_scrub_begin_week_day 0 2024-09-18T16:39:56.462 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:39:56.463 INFO:tasks.workunit.client.0.smithi046.stdout: "success": "osd_scrub_begin_week_day = '' (not observed, change may require restart) " 2024-09-18T16:39:56.463 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:39:56.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:248: TEST_scrub_extended_sleep: ceph tell osd.1 config set osd_scrub_end_week_day 0 2024-09-18T16:39:56.607 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:39:56.608 INFO:tasks.workunit.client.0.smithi046.stdout: "success": "osd_scrub_end_week_day = '' (not observed, change may require restart) " 2024-09-18T16:39:56.608 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:39:56.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:252: TEST_scrub_extended_sleep: count=0 2024-09-18T16:39:56.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:253: TEST_scrub_extended_sleep: PASSED=false 2024-09-18T16:39:56.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i=0 )) 2024-09-18T16:39:56.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i < 25 )) 2024-09-18T16:39:56.619 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: expr 0 + 1 2024-09-18T16:39:56.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: count=1 2024-09-18T16:39:56.622 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: get_last_scrub_stamp 1.0 2024-09-18T16:39:56.622 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:39:56.622 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:39:56.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:39:56.623 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:39:56.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: test 2024-09-18T16:39:47.481174+0000 '>' 2024-09-18T16:39:47.481174+0000 2024-09-18T16:39:56.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:265: TEST_scrub_extended_sleep: sleep 1 2024-09-18T16:39:57.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i++ )) 2024-09-18T16:39:57.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i < 25 )) 2024-09-18T16:39:57.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: expr 1 + 1 2024-09-18T16:39:57.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: count=2 2024-09-18T16:39:57.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: get_last_scrub_stamp 1.0 2024-09-18T16:39:57.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:39:57.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:39:57.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:39:57.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:39:58.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: test 2024-09-18T16:39:47.481174+0000 '>' 2024-09-18T16:39:47.481174+0000 2024-09-18T16:39:58.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:265: TEST_scrub_extended_sleep: sleep 1 2024-09-18T16:39:59.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i++ )) 2024-09-18T16:39:59.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i < 25 )) 2024-09-18T16:39:59.239 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: expr 2 + 1 2024-09-18T16:39:59.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: count=3 2024-09-18T16:39:59.242 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: get_last_scrub_stamp 1.0 2024-09-18T16:39:59.242 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:39:59.242 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:39:59.243 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:39:59.243 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:39:59.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: test 2024-09-18T16:39:47.481174+0000 '>' 2024-09-18T16:39:47.481174+0000 2024-09-18T16:39:59.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:265: TEST_scrub_extended_sleep: sleep 1 2024-09-18T16:40:00.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i++ )) 2024-09-18T16:40:00.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i < 25 )) 2024-09-18T16:40:00.548 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: expr 3 + 1 2024-09-18T16:40:00.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: count=4 2024-09-18T16:40:00.551 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: get_last_scrub_stamp 1.0 2024-09-18T16:40:00.551 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:40:00.551 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:40:00.551 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:40:00.552 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:40:00.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: test 2024-09-11T16:38:14.301057+0000 '>' 2024-09-18T16:39:47.481174+0000 2024-09-18T16:40:00.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:265: TEST_scrub_extended_sleep: sleep 1 2024-09-18T16:40:01.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i++ )) 2024-09-18T16:40:01.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i < 25 )) 2024-09-18T16:40:01.860 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: expr 4 + 1 2024-09-18T16:40:01.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: count=5 2024-09-18T16:40:01.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: get_last_scrub_stamp 1.0 2024-09-18T16:40:01.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:40:01.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:40:01.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:40:01.864 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:40:02.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: test 2024-09-11T16:38:14.301057+0000 '>' 2024-09-18T16:39:47.481174+0000 2024-09-18T16:40:02.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:265: TEST_scrub_extended_sleep: sleep 1 2024-09-18T16:40:03.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i++ )) 2024-09-18T16:40:03.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i < 25 )) 2024-09-18T16:40:03.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: expr 5 + 1 2024-09-18T16:40:03.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: count=6 2024-09-18T16:40:03.174 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: get_last_scrub_stamp 1.0 2024-09-18T16:40:03.174 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:40:03.174 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:40:03.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:40:03.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:40:03.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: test 2024-09-11T16:38:14.301057+0000 '>' 2024-09-18T16:39:47.481174+0000 2024-09-18T16:40:03.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:265: TEST_scrub_extended_sleep: sleep 1 2024-09-18T16:40:04.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i++ )) 2024-09-18T16:40:04.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i < 25 )) 2024-09-18T16:40:04.478 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: expr 6 + 1 2024-09-18T16:40:04.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: count=7 2024-09-18T16:40:04.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: get_last_scrub_stamp 1.0 2024-09-18T16:40:04.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:40:04.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:40:04.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:40:04.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:40:04.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: test 2024-09-11T16:38:14.301057+0000 '>' 2024-09-18T16:39:47.481174+0000 2024-09-18T16:40:04.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:265: TEST_scrub_extended_sleep: sleep 1 2024-09-18T16:40:05.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i++ )) 2024-09-18T16:40:05.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i < 25 )) 2024-09-18T16:40:05.791 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: expr 7 + 1 2024-09-18T16:40:05.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: count=8 2024-09-18T16:40:05.794 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: get_last_scrub_stamp 1.0 2024-09-18T16:40:05.794 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:40:05.794 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:40:05.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:40:05.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:40:06.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: test 2024-09-11T16:38:14.301057+0000 '>' 2024-09-18T16:39:47.481174+0000 2024-09-18T16:40:06.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:265: TEST_scrub_extended_sleep: sleep 1 2024-09-18T16:40:07.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i++ )) 2024-09-18T16:40:07.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i < 25 )) 2024-09-18T16:40:07.098 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: expr 8 + 1 2024-09-18T16:40:07.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: count=9 2024-09-18T16:40:07.101 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: get_last_scrub_stamp 1.0 2024-09-18T16:40:07.101 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:40:07.101 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:40:07.102 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:40:07.102 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:40:07.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: test 2024-09-11T16:38:14.301057+0000 '>' 2024-09-18T16:39:47.481174+0000 2024-09-18T16:40:07.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:265: TEST_scrub_extended_sleep: sleep 1 2024-09-18T16:40:08.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i++ )) 2024-09-18T16:40:08.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i < 25 )) 2024-09-18T16:40:08.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: expr 9 + 1 2024-09-18T16:40:08.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: count=10 2024-09-18T16:40:08.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: get_last_scrub_stamp 1.0 2024-09-18T16:40:08.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:40:08.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:40:08.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:40:08.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:40:08.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: test 2024-09-11T16:38:14.301057+0000 '>' 2024-09-18T16:39:47.481174+0000 2024-09-18T16:40:08.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:265: TEST_scrub_extended_sleep: sleep 1 2024-09-18T16:40:09.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i++ )) 2024-09-18T16:40:09.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i < 25 )) 2024-09-18T16:40:09.735 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: expr 10 + 1 2024-09-18T16:40:09.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: count=11 2024-09-18T16:40:09.738 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: get_last_scrub_stamp 1.0 2024-09-18T16:40:09.738 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:40:09.738 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:40:09.738 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:40:09.739 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:40:10.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: test 2024-09-11T16:38:14.301057+0000 '>' 2024-09-18T16:39:47.481174+0000 2024-09-18T16:40:10.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:265: TEST_scrub_extended_sleep: sleep 1 2024-09-18T16:40:11.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i++ )) 2024-09-18T16:40:11.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i < 25 )) 2024-09-18T16:40:11.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: expr 11 + 1 2024-09-18T16:40:11.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: count=12 2024-09-18T16:40:11.052 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: get_last_scrub_stamp 1.0 2024-09-18T16:40:11.052 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:40:11.052 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:40:11.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:40:11.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:40:11.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: test 2024-09-11T16:38:14.301057+0000 '>' 2024-09-18T16:39:47.481174+0000 2024-09-18T16:40:11.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:265: TEST_scrub_extended_sleep: sleep 1 2024-09-18T16:40:12.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i++ )) 2024-09-18T16:40:12.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i < 25 )) 2024-09-18T16:40:12.359 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: expr 12 + 1 2024-09-18T16:40:12.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: count=13 2024-09-18T16:40:12.362 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: get_last_scrub_stamp 1.0 2024-09-18T16:40:12.362 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:40:12.362 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:40:12.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:40:12.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:40:12.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: test 2024-09-11T16:38:14.301057+0000 '>' 2024-09-18T16:39:47.481174+0000 2024-09-18T16:40:12.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:265: TEST_scrub_extended_sleep: sleep 1 2024-09-18T16:40:13.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i++ )) 2024-09-18T16:40:13.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i < 25 )) 2024-09-18T16:40:13.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: expr 13 + 1 2024-09-18T16:40:13.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: count=14 2024-09-18T16:40:13.675 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: get_last_scrub_stamp 1.0 2024-09-18T16:40:13.676 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:40:13.676 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:40:13.676 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:40:13.676 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:40:13.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: test 2024-09-11T16:38:14.301057+0000 '>' 2024-09-18T16:39:47.481174+0000 2024-09-18T16:40:13.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:265: TEST_scrub_extended_sleep: sleep 1 2024-09-18T16:40:14.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i++ )) 2024-09-18T16:40:14.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i < 25 )) 2024-09-18T16:40:14.983 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: expr 14 + 1 2024-09-18T16:40:14.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: count=15 2024-09-18T16:40:14.986 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: get_last_scrub_stamp 1.0 2024-09-18T16:40:14.986 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:40:14.986 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:40:14.987 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:40:14.987 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:40:15.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: test 2024-09-11T16:38:14.301057+0000 '>' 2024-09-18T16:39:47.481174+0000 2024-09-18T16:40:15.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:265: TEST_scrub_extended_sleep: sleep 1 2024-09-18T16:40:16.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i++ )) 2024-09-18T16:40:16.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i < 25 )) 2024-09-18T16:40:16.288 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: expr 15 + 1 2024-09-18T16:40:16.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: count=16 2024-09-18T16:40:16.290 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: get_last_scrub_stamp 1.0 2024-09-18T16:40:16.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:40:16.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:40:16.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:40:16.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:40:16.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: test 2024-09-11T16:38:14.301057+0000 '>' 2024-09-18T16:39:47.481174+0000 2024-09-18T16:40:16.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:265: TEST_scrub_extended_sleep: sleep 1 2024-09-18T16:40:17.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i++ )) 2024-09-18T16:40:17.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i < 25 )) 2024-09-18T16:40:17.598 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: expr 16 + 1 2024-09-18T16:40:17.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: count=17 2024-09-18T16:40:17.601 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: get_last_scrub_stamp 1.0 2024-09-18T16:40:17.601 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:40:17.601 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:40:17.602 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:40:17.602 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:40:17.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: test 2024-09-11T16:38:14.301057+0000 '>' 2024-09-18T16:39:47.481174+0000 2024-09-18T16:40:17.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:265: TEST_scrub_extended_sleep: sleep 1 2024-09-18T16:40:18.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i++ )) 2024-09-18T16:40:18.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i < 25 )) 2024-09-18T16:40:18.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: expr 17 + 1 2024-09-18T16:40:18.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: count=18 2024-09-18T16:40:18.914 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: get_last_scrub_stamp 1.0 2024-09-18T16:40:18.914 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:40:18.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:40:18.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:40:18.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:40:19.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: test 2024-09-11T16:38:14.301057+0000 '>' 2024-09-18T16:39:47.481174+0000 2024-09-18T16:40:19.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:265: TEST_scrub_extended_sleep: sleep 1 2024-09-18T16:40:20.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i++ )) 2024-09-18T16:40:20.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:254: TEST_scrub_extended_sleep: (( i < 25 )) 2024-09-18T16:40:20.219 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: expr 18 + 1 2024-09-18T16:40:20.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:255: TEST_scrub_extended_sleep: count=19 2024-09-18T16:40:20.221 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: get_last_scrub_stamp 1.0 2024-09-18T16:40:20.222 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:40:20.222 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:40:20.222 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:40:20.222 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:40:20.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:256: TEST_scrub_extended_sleep: test 2024-09-18T16:40:15.408577+0000 '>' 2024-09-18T16:39:47.481174+0000 2024-09-18T16:40:20.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:258: TEST_scrub_extended_sleep: '[' 19 -lt 10 ']' 2024-09-18T16:40:20.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:262: TEST_scrub_extended_sleep: PASSED=true 2024-09-18T16:40:20.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:263: TEST_scrub_extended_sleep: break 2024-09-18T16:40:20.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:269: TEST_scrub_extended_sleep: '[' true = false ']' 2024-09-18T16:40:20.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: teardown td/osd-scrub-test 2024-09-18T16:40:20.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-09-18T16:40:20.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:40:20.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-18T16:40:20.534 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:40:20.534 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:40:20.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:40:20.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:40:20.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:40:20.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:40:20.652 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:40:20.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:40:20.654 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:40:20.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:40:20.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:40:20.657 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:40:20.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:40:20.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:40:20.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:40:20.660 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:40:20.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:40:20.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:40:20.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-09-18T16:40:20.684 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:40:20.684 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:40:20.684 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:40:20.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.562265 2024-09-18T16:40:20.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:40:20.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:40:20.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-09-18T16:40:20.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: setup td/osd-scrub-test 2024-09-18T16:40:20.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-test 2024-09-18T16:40:20.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-test 2024-09-18T16:40:20.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-09-18T16:40:20.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:40:20.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-18T16:40:20.688 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:40:20.688 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:40:20.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:40:20.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:40:20.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:40:20.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:40:20.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:40:20.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:40:20.695 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:40:20.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:40:20.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:40:20.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:40:20.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:40:20.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:40:20.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:40:20.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:40:20.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:40:20.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:40:20.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-09-18T16:40:20.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:40:20.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:40:20.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:40:20.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.562265 2024-09-18T16:40:20.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:40:20.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:40:20.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-test 2024-09-18T16:40:20.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T16:40:20.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:40:20.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:40:20.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.562265 2024-09-18T16:40:20.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T16:40:20.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T16:40:20.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T16:40:20.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-09-18T16:40:20.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: TEST_scrub_permit_time td/osd-scrub-test 2024-09-18T16:40:20.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:401: TEST_scrub_permit_time: local dir=td/osd-scrub-test 2024-09-18T16:40:20.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:402: TEST_scrub_permit_time: local poolname=test 2024-09-18T16:40:20.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:403: TEST_scrub_permit_time: local OSDS=3 2024-09-18T16:40:20.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:404: TEST_scrub_permit_time: local objects=15 2024-09-18T16:40:20.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:406: TEST_scrub_permit_time: TESTDATA=testdata.562265 2024-09-18T16:40:20.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:408: TEST_scrub_permit_time: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2024-09-18T16:40:20.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-test 2024-09-18T16:40:20.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T16:40:20.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T16:40:20.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T16:40:20.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-test/a 2024-09-18T16:40:20.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2024-09-18T16:40:21.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T16:40:21.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:40:21.086 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:40:21.086 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:40:21.086 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:40:21.086 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:40:21.087 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:40:21.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-18T16:40:21.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T16:40:21.376 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T16:40:21.377 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:40:21.377 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:40:21.377 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T16:40:21.377 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T16:40:21.378 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:40:21.378 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:40:21.378 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:40:21.379 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:40:21.379 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:40:21.379 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:40:21.379 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.562265/ceph-mon.a.asok 2024-09-18T16:40:21.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:40:21.380 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.562265/ceph-mon.a.asok config get fsid 2024-09-18T16:40:21.478 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T16:40:21.478 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:40:21.478 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:40:21.478 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T16:40:21.478 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T16:40:21.479 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:40:21.479 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:40:21.479 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:40:21.480 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:40:21.480 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:40:21.480 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:40:21.480 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.562265/ceph-mon.a.asok 2024-09-18T16:40:21.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:40:21.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.562265/ceph-mon.a.asok config get mon_host 2024-09-18T16:40:21.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:409: TEST_scrub_permit_time: run_mgr td/osd-scrub-test x 2024-09-18T16:40:21.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-test 2024-09-18T16:40:21.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T16:40:21.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T16:40:21.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T16:40:21.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-test/x 2024-09-18T16:40:21.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T16:40:22.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T16:40:22.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:40:22.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:40:22.435 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:40:22.435 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:40:22.435 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:40:22.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:40:22.437 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:40:22.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:40:22.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:410: TEST_scrub_permit_time: sed 's/^0//' 2024-09-18T16:40:22.476 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:410: TEST_scrub_permit_time: date -d '2 hour ago' +%H 2024-09-18T16:40:22.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:410: TEST_scrub_permit_time: local scrub_begin_hour=14 2024-09-18T16:40:22.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:411: TEST_scrub_permit_time: date -d '1 hour ago' +%H 2024-09-18T16:40:22.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:411: TEST_scrub_permit_time: sed 's/^0//' 2024-09-18T16:40:22.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:411: TEST_scrub_permit_time: local scrub_end_hour=15 2024-09-18T16:40:22.483 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:412: TEST_scrub_permit_time: expr 3 - 1 2024-09-18T16:40:22.484 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:412: TEST_scrub_permit_time: seq 0 2 2024-09-18T16:40:22.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:412: TEST_scrub_permit_time: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:40:22.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:414: TEST_scrub_permit_time: run_osd td/osd-scrub-test 0 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=14 --osd_scrub_end_hour=15 2024-09-18T16:40:22.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-09-18T16:40:22.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:40:22.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T16:40:22.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:40:22.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/0 2024-09-18T16:40:22.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-18T16:40:22.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:40:22.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:40:22.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:40:22.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-09-18T16:40:22.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-09-18T16:40:22.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:40:22.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:40:22.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-18T16:40:22.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:40:22.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:40:22.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:40:22.489 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:40:22.489 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:40:22.489 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:40:22.491 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:40:22.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:40:22.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:40:22.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:40:22.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:40:22.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-18T16:40:22.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-18T16:40:22.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:40:22.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:40:22.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:40:22.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:40:22.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:40:22.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=14 --osd_scrub_end_hour=15' 2024-09-18T16:40:22.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/0 2024-09-18T16:40:22.492 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:40:22.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ce06224a-3bba-40ff-bb93-50717a37adb5 2024-09-18T16:40:22.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ce06224a-3bba-40ff-bb93-50717a37adb5' 2024-09-18T16:40:22.493 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 ce06224a-3bba-40ff-bb93-50717a37adb5 2024-09-18T16:40:22.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:40:22.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB2Autm8AZQHhAASFIBdaVTGCIEgLY+pATlcw== 2024-09-18T16:40:22.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB2Autm8AZQHhAASFIBdaVTGCIEgLY+pATlcw=="}' 2024-09-18T16:40:22.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ce06224a-3bba-40ff-bb93-50717a37adb5 -i td/osd-scrub-test/0/new.json 2024-09-18T16:40:22.812 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:40:22.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/0/new.json 2024-09-18T16:40:22.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=14 --osd_scrub_end_hour=15 --mkfs --key AQB2Autm8AZQHhAASFIBdaVTGCIEgLY+pATlcw== --osd-uuid ce06224a-3bba-40ff-bb93-50717a37adb5 2024-09-18T16:40:22.857 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:40:22.852+0000 7f752e1ad540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:40:22.904 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:40:22.896+0000 7f752e1ad540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:40:22.905 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:40:22.900+0000 7f752e1ad540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:40:22.906 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:40:22.900+0000 7f752e1ad540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:40:22.906 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:40:22.900+0000 7f752e1ad540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:40:22.906 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:40:22.900+0000 7f752e1ad540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:40:22.950 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:40:22.944+0000 7f752e1ad540 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-09-18T16:40:29.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-09-18T16:40:29.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:40:29.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T16:40:29.277 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T16:40:29.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:40:29.660 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T16:40:29.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T16:40:29.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=14 --osd_scrub_end_hour=15 2024-09-18T16:40:29.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:40:29.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:40:29.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:40:29.690 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:40:29.684+0000 7f33ae619540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:40:29.693 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:40:29.688+0000 7f33ae619540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:40:29.694 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:40:29.688+0000 7f33ae619540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:40:29.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T16:40:29.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:40:29.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:40:29.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:40:29.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:40:29.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:40:29.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:40:29.965 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:40:29.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:40:29.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:40:30.251 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:40:30.244+0000 7f33ae619540 -1 Falling back to public interface 2024-09-18T16:40:30.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:40:31.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:40:31.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:40:31.285 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:40:31.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:40:31.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:40:31.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:40:31.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:40:32.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:40:32.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:40:32.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:40:32.601 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:40:32.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:40:32.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:40:32.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:40:33.021 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:40:33.016+0000 7f33ae619540 -1 osd.0 0 log_to_monitors true 2024-09-18T16:40:33.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:40:33.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:40:33.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:40:33.906 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:40:33.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:40:33.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:40:34.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:40:35.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:40:35.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:40:35.224 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:40:35.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:40:35.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:40:35.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:40:35.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:40:36.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:40:36.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:40:36.534 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:40:36.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:40:36.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:40:36.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:40:36.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:40:37.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:40:37.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:40:37.858 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:40:37.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:40:37.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:40:37.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:40:38.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:40:39.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:40:39.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:40:39.170 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:40:39.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:40:39.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:40:39.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:40:39.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:40:40.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:40:40.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:40:40.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:40:40.483 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:40:40.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:40:40.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:40:40.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:40:41.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:40:41.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:40:41.918 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:40:41.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:40:41.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:40:41.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:40:42.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:40:43.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:40:43.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:40:43.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:40:43.223 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:40:43.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:40:43.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:40:43.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:40:44.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:40:44.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:40:44.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:40:44.536 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:40:44.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:40:44.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:40:44.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:40:45.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:40:45.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:40:45.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:40:45.862 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:40:45.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:40:45.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:40:46.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:40:47.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:40:47.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:40:47.317 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:40:47.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:40:47.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:40:47.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:40:47.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:40:48.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:40:48.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:40:48.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:40:48.683 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:40:48.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:40:48.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:40:49.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:40:50.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:40:50.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:40:50.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:40:50.405 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:40:50.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:40:50.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:40:51.113 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:40:52.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:40:52.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:40:52.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:40:52.116 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:40:52.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:40:52.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:40:52.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:40:53.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:40:53.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:40:53.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:40:53.632 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:40:53.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:40:53.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:40:54.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:40:55.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:40:55.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:40:55.363 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:40:55.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:40:55.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:40:55.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:40:55.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:40:56.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:40:56.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:40:56.674 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T16:40:56.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T16:40:56.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:40:56.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:40:56.985 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/655217,v1:127.0.0.1:6803/655217] [v2:127.0.0.1:6804/655217,v1:127.0.0.1:6805/655217] exists,up ce06224a-3bba-40ff-bb93-50717a37adb5 2024-09-18T16:40:56.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:40:56.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:40:56.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:40:56.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:412: TEST_scrub_permit_time: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:40:56.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:414: TEST_scrub_permit_time: run_osd td/osd-scrub-test 1 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=14 --osd_scrub_end_hour=15 2024-09-18T16:40:56.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-09-18T16:40:56.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:40:56.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T16:40:56.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:40:56.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/1 2024-09-18T16:40:56.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-18T16:40:56.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:40:56.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:40:56.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:40:56.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-09-18T16:40:56.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-09-18T16:40:56.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:40:56.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:40:56.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-18T16:40:56.990 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:40:56.990 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:40:56.990 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:40:56.992 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:40:56.992 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:40:56.992 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:40:56.992 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:40:56.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:40:56.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:40:56.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:40:56.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:40:56.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-18T16:40:56.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-18T16:40:56.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:40:56.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:40:56.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:40:56.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:40:56.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:40:56.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=14 --osd_scrub_end_hour=15' 2024-09-18T16:40:56.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/1 2024-09-18T16:40:56.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:40:56.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9ae006df-215b-43be-8a82-b03fb55bd804 2024-09-18T16:40:56.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9ae006df-215b-43be-8a82-b03fb55bd804' 2024-09-18T16:40:56.994 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 9ae006df-215b-43be-8a82-b03fb55bd804 2024-09-18T16:40:56.995 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:40:57.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCZAutmf+a9ABAAeI7I1LOJa/m9jhkZ9aygCA== 2024-09-18T16:40:57.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCZAutmf+a9ABAAeI7I1LOJa/m9jhkZ9aygCA=="}' 2024-09-18T16:40:57.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9ae006df-215b-43be-8a82-b03fb55bd804 -i td/osd-scrub-test/1/new.json 2024-09-18T16:40:57.405 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:40:57.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/1/new.json 2024-09-18T16:40:57.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=14 --osd_scrub_end_hour=15 --mkfs --key AQCZAutmf+a9ABAAeI7I1LOJa/m9jhkZ9aygCA== --osd-uuid 9ae006df-215b-43be-8a82-b03fb55bd804 2024-09-18T16:40:57.451 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:40:57.444+0000 7fa0b813c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:40:57.455 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:40:57.448+0000 7fa0b813c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:40:57.456 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:40:57.448+0000 7fa0b813c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:40:57.457 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:40:57.452+0000 7fa0b813c540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:40:57.457 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:40:57.452+0000 7fa0b813c540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:40:57.457 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:40:57.452+0000 7fa0b813c540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:40:57.524 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:40:57.516+0000 7fa0b813c540 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-09-18T16:41:04.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-09-18T16:41:04.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:41:04.010 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T16:41:04.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T16:41:04.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:41:04.388 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T16:41:04.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T16:41:04.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:41:04.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=14 --osd_scrub_end_hour=15 2024-09-18T16:41:04.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:41:04.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:41:04.419 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:41:04.412+0000 7f38175e4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:41:04.422 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:41:04.416+0000 7f38175e4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:41:04.425 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:41:04.416+0000 7f38175e4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:41:04.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T16:41:04.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:41:04.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T16:41:04.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:41:04.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:41:04.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:04.706 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:41:04.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:41:04.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:04.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:41:04.971 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:41:04.964+0000 7f38175e4540 -1 Falling back to public interface 2024-09-18T16:41:05.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:06.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:06.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:06.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:41:06.050 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:41:06.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:06.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:41:06.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:07.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:07.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:07.359 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:41:07.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:41:07.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:07.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:41:07.667 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:41:07.660+0000 7f38175e4540 -1 osd.1 0 log_to_monitors true 2024-09-18T16:41:07.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:08.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:08.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:08.680 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:41:08.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:41:08.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:08.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:41:09.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:10.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:10.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:10.026 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:41:10.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:41:10.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:10.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:41:10.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:11.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:11.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:11.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:41:11.350 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:41:11.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:11.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:41:11.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:12.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:12.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:12.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:41:12.731 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:41:12.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:12.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:41:13.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:14.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:14.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:14.040 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:41:14.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:41:14.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:14.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:41:14.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:15.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:15.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:15.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:41:15.359 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:41:15.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:15.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:41:15.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:16.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:16.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:16.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:41:16.754 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:41:16.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:16.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:41:17.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:18.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:18.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:18.073 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:41:18.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:41:18.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:18.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:41:18.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:19.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:19.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:19.379 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:41:19.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:41:19.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:19.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:41:19.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:20.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:20.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:20.703 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:41:20.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:41:20.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:20.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:41:21.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:22.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:22.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:22.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:41:22.424 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:41:22.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:22.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:41:23.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:24.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:24.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:24.192 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:41:24.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:41:24.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:24.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:41:24.506 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:25.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:25.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:25.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:41:25.509 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:41:25.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:25.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:41:26.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:27.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:27.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:27.054 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:41:27.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:41:27.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:27.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:41:27.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:28.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:28.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:28.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:41:28.734 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:41:28.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:28.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:41:29.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:30.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:30.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:30.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:41:30.083 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:41:30.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:30.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:41:30.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:31.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:31.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:31.421 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T16:41:31.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T16:41:31.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:31.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:41:31.747 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/657021,v1:127.0.0.1:6811/657021] [v2:127.0.0.1:6812/657021,v1:127.0.0.1:6813/657021] exists,up 9ae006df-215b-43be-8a82-b03fb55bd804 2024-09-18T16:41:31.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:41:31.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:41:31.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:41:31.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:412: TEST_scrub_permit_time: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:41:31.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:414: TEST_scrub_permit_time: run_osd td/osd-scrub-test 2 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=14 --osd_scrub_end_hour=15 2024-09-18T16:41:31.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-09-18T16:41:31.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:41:31.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T16:41:31.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:41:31.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/2 2024-09-18T16:41:31.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-18T16:41:31.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:41:31.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:41:31.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:41:31.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2024-09-18T16:41:31.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2024-09-18T16:41:31.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:41:31.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:41:31.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-18T16:41:31.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:41:31.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:41:31.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:41:31.752 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:41:31.753 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:41:31.753 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:41:31.753 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:41:31.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:41:31.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:41:31.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:41:31.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:41:31.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-18T16:41:31.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-18T16:41:31.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:41:31.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:41:31.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:41:31.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:41:31.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:41:31.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=14 --osd_scrub_end_hour=15' 2024-09-18T16:41:31.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/2 2024-09-18T16:41:31.754 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:41:31.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7f11e10b-a7a8-408a-b50c-fa618e056efb 2024-09-18T16:41:31.756 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 7f11e10b-a7a8-408a-b50c-fa618e056efb 2024-09-18T16:41:31.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 7f11e10b-a7a8-408a-b50c-fa618e056efb' 2024-09-18T16:41:31.756 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:41:31.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC7Autm7gQHLhAAsX/3ZeY3FS1rXuuWkKFdww== 2024-09-18T16:41:31.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC7Autm7gQHLhAAsX/3ZeY3FS1rXuuWkKFdww=="}' 2024-09-18T16:41:31.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7f11e10b-a7a8-408a-b50c-fa618e056efb -i td/osd-scrub-test/2/new.json 2024-09-18T16:41:32.169 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:41:32.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/2/new.json 2024-09-18T16:41:32.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=14 --osd_scrub_end_hour=15 --mkfs --key AQC7Autm7gQHLhAAsX/3ZeY3FS1rXuuWkKFdww== --osd-uuid 7f11e10b-a7a8-408a-b50c-fa618e056efb 2024-09-18T16:41:32.225 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:41:32.223+0000 7fef406ae540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:41:32.235 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:41:32.231+0000 7fef406ae540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:41:32.238 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:41:32.235+0000 7fef406ae540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:41:32.239 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:41:32.235+0000 7fef406ae540 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-09-18T16:41:32.239 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:41:32.235+0000 7fef406ae540 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-09-18T16:41:32.239 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:41:32.235+0000 7fef406ae540 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-09-18T16:41:32.306 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:41:32.303+0000 7fef406ae540 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-09-18T16:41:38.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2024-09-18T16:41:38.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:41:38.875 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T16:41:38.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T16:41:38.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:41:39.260 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T16:41:39.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T16:41:39.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=14 --osd_scrub_end_hour=15 2024-09-18T16:41:39.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:41:39.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:41:39.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:41:39.292 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:41:39.287+0000 7f942f349540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:41:39.294 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:41:39.291+0000 7f942f349540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:41:39.300 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:41:39.291+0000 7f942f349540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:41:39.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T16:41:39.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:41:39.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T16:41:39.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:41:39.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:41:39.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:39.602 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:41:39.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:41:39.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:39.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:41:39.835 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:41:39.831+0000 7f942f349540 -1 Falling back to public interface 2024-09-18T16:41:39.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:40.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:40.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:40.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:41:40.908 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:41:40.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:40.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:41:41.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:42.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:42.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:42.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:41:42.224 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:41:42.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:42.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:41:42.365 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:41:42.363+0000 7f942f349540 -1 osd.2 0 log_to_monitors true 2024-09-18T16:41:42.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:43.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:43.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:43.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:41:43.586 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:41:43.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:43.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:41:43.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:44.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:44.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:44.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:41:44.963 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:41:44.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:44.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:41:45.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:46.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:46.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:46.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:41:46.271 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:41:46.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:46.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:41:46.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:47.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:47.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:47.624 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:41:47.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:41:47.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:47.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:41:47.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:48.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:48.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:48.983 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:41:48.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:41:48.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:48.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:41:49.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:50.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:50.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:50.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:41:50.288 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:41:50.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:50.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:41:50.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:51.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:51.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:51.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:41:51.625 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:41:51.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:51.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:41:51.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:52.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:52.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:52.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:41:52.958 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:41:52.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:52.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:41:53.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:54.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:54.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:54.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:41:54.265 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:41:54.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:54.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:41:54.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:55.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:55.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:55.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:41:55.888 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:41:55.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:55.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:41:56.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:57.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:57.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:57.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:41:57.519 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:41:57.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:57.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:41:57.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:41:58.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:41:58.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:41:58.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:41:58.834 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:41:58.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:41:58.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:41:59.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:42:00.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:42:00.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:42:00.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:42:00.143 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:42:00.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:42:00.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:42:00.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:42:01.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:42:01.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:42:01.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:42:01.627 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:42:01.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:42:01.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:42:02.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:42:03.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:42:03.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:42:03.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:42:03.018 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:42:03.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:42:03.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:42:03.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:42:04.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:42:04.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:42:04.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:42:04.680 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:42:04.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:42:04.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:42:04.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:42:05.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:42:05.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:42:05.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T16:42:05.995 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T16:42:05.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:42:05.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:42:06.300 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/658825,v1:127.0.0.1:6819/658825] [v2:127.0.0.1:6820/658825,v1:127.0.0.1:6821/658825] exists,up 7f11e10b-a7a8-408a-b50c-fa618e056efb 2024-09-18T16:42:06.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:42:06.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:42:06.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:42:06.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:422: TEST_scrub_permit_time: create_pool test 1 1 2024-09-18T16:42:06.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-18T16:42:06.948 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'test' created 2024-09-18T16:42:06.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T16:42:07.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:423: TEST_scrub_permit_time: wait_for_clean 2024-09-18T16:42:07.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:42:07.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:42:07.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:42:07.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:42:07.962 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:42:07.963 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:42:07.963 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:42:07.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:42:07.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:42:07.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:42:08.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:42:08.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:42:08.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:42:08.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=500 2024-09-18T16:42:08.188 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:42:08.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:42:08.557 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:42:08.558 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:42:08.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:42:08.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:42:08.558 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:42:08.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836501 2024-09-18T16:42:08.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836501 2024-09-18T16:42:08.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836501' 2024-09-18T16:42:08.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:42:08.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:42:08.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672976 2024-09-18T16:42:08.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672976 2024-09-18T16:42:08.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836501 1-42949672976' 2024-09-18T16:42:08.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:42:08.827 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:42:08.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509448 2024-09-18T16:42:08.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509448 2024-09-18T16:42:08.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836501 1-42949672976 2-64424509448' 2024-09-18T16:42:08.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:42:08.984 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836501 2024-09-18T16:42:08.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:42:08.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:42:08.988 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836501 2024-09-18T16:42:08.988 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:42:08.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836501 2024-09-18T16:42:08.990 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836501 2024-09-18T16:42:08.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-18T16:42:08.991 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:42:09.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836500 -lt 21474836501 2024-09-18T16:42:09.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:42:10.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-18T16:42:10.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:42:10.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836501 -lt 21474836501 2024-09-18T16:42:10.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:42:10.661 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672976 2024-09-18T16:42:10.661 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:42:10.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:42:10.664 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672976 2024-09-18T16:42:10.664 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:42:10.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672976 2024-09-18T16:42:10.666 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672976 2024-09-18T16:42:10.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-18T16:42:10.666 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:42:11.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672976 -lt 42949672976 2024-09-18T16:42:11.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:42:11.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509448 2024-09-18T16:42:11.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:42:11.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:42:11.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:42:11.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509448 2024-09-18T16:42:11.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509448 2024-09-18T16:42:11.030 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509448 2024-09-18T16:42:11.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-18T16:42:11.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:42:11.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509448 -lt 64424509448 2024-09-18T16:42:11.369 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:42:11.370 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:42:11.370 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:42:11.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:42:11.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:42:11.735 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:42:11.735 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:42:11.736 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:42:11.736 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:42:11.736 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:42:11.736 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:42:12.103 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:42:12.104 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:42:12.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:42:12.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:42:12.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:42:12.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:42:12.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:42:12.477 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:426: TEST_scrub_permit_time: get_pg test SOMETHING 2024-09-18T16:42:12.477 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-18T16:42:12.477 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-09-18T16:42:12.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test SOMETHING 2024-09-18T16:42:12.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T16:42:12.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:426: TEST_scrub_permit_time: local pgid=1.0 2024-09-18T16:42:12.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:427: TEST_scrub_permit_time: get_primary test SOMETHING 2024-09-18T16:42:12.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T16:42:12.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-09-18T16:42:12.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test SOMETHING 2024-09-18T16:42:12.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T16:42:13.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:427: TEST_scrub_permit_time: local primary=1 2024-09-18T16:42:13.157 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:428: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:13.158 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:13.158 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:13.158 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:13.158 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:13.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:428: TEST_scrub_permit_time: local last_scrub=2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:13.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:433: TEST_scrub_permit_time: ceph tell 1.0 scrub 86400 2024-09-18T16:42:13.681 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:42:13.681 INFO:tasks.workunit.client.0.smithi046.stdout: "deep": false, 2024-09-18T16:42:13.682 INFO:tasks.workunit.client.0.smithi046.stdout: "stamp": "2024-09-17T16:40:33.680699+0000" 2024-09-18T16:42:13.682 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:42:13.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i=0 )) 2024-09-18T16:42:13.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:13.695 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:13.695 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:13.695 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:13.695 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:13.695 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:13.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-18T16:42:06.760992+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:13.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:14.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:14.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:14.991 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:14.991 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:14.991 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:14.992 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:14.992 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:15.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-18T16:42:06.760992+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:15.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:16.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:16.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:16.303 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:16.303 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:16.303 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:16.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:16.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:16.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-18T16:42:06.760992+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:16.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:17.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:17.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:17.610 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:17.610 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:17.610 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:17.611 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:17.611 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:17.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:17.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:18.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:18.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:18.916 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:18.916 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:18.916 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:18.916 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:18.917 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:19.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:19.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:20.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:20.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:20.232 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:20.232 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:20.232 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:20.233 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:20.233 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:20.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:20.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:21.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:21.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:21.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:21.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:21.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:21.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:21.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:21.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:21.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:22.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:22.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:22.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:22.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:22.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:22.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:22.866 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:23.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:23.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:24.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:24.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:24.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:24.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:24.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:24.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:24.162 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:24.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:24.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:25.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:25.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:25.468 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:25.468 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:25.468 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:25.468 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:25.469 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:25.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:25.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:26.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:26.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:26.775 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:26.775 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:26.775 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:26.775 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:26.776 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:27.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:27.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:28.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:28.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:28.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:28.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:28.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:28.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:28.097 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:28.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:28.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:29.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:29.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:29.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:29.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:29.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:29.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:29.412 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:29.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:29.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:30.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:30.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:30.718 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:30.718 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:30.718 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:30.719 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:30.719 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:31.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:31.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:32.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:32.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:32.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:32.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:32.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:32.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:32.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:32.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:32.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:33.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:33.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:33.338 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:33.338 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:33.339 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:33.339 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:33.339 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:33.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:33.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:34.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:34.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:34.649 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:34.649 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:34.649 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:34.650 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:34.650 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:34.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:34.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:35.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:35.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:35.963 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:35.963 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:35.963 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:35.963 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:35.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:36.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:36.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:37.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:37.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:37.270 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:37.270 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:37.270 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:37.271 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:37.271 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:37.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:37.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:38.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:38.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:38.578 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:38.579 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:38.579 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:38.579 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:38.579 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:38.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:38.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:39.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:39.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:39.893 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:39.893 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:39.893 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:39.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:39.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:40.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:40.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:41.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:41.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:41.206 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:41.206 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:41.206 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:41.206 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:41.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:41.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:41.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:42.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:42.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:42.511 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:42.511 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:42.511 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:42.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:42.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:42.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:42.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:43.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:43.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:43.873 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:43.873 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:43.873 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:43.873 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:43.873 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:44.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:44.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:45.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:45.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:45.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:45.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:45.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:45.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:45.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:45.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:45.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:46.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:46.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:46.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:46.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:46.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:46.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:46.494 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:46.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:46.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:47.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:47.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:47.807 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:47.807 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:47.807 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:47.807 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:47.808 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:48.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:48.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:49.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:49.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:49.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:49.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:49.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:49.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:49.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:49.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:49.436 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:50.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:50.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:50.439 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:50.439 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:50.439 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:50.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:50.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:50.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:50.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:51.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:51.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:51.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-18T16:42:51.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:42:51.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:42:51.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:42:51.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:42:52.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:437: TEST_scrub_permit_time: test 2024-09-17T16:40:33.680699+0000 '>' 2024-09-18T16:42:06.760992+0000 2024-09-18T16:42:52.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: sleep 1 2024-09-18T16:42:53.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i++ )) 2024-09-18T16:42:53.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: (( i < 30 )) 2024-09-18T16:42:53.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: teardown td/osd-scrub-test 2024-09-18T16:42:53.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-09-18T16:42:53.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:42:53.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-18T16:42:53.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:42:53.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:42:53.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:42:53.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:42:53.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:42:53.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:42:53.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:42:53.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:42:53.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:42:53.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:42:53.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:42:53.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:42:53.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:42:53.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:42:53.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:42:53.388 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:42:53.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:42:53.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:42:53.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-09-18T16:42:53.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:42:53.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:42:53.413 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:42:53.413 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.562265 2024-09-18T16:42:53.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:42:53.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:42:53.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-09-18T16:42:53.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: setup td/osd-scrub-test 2024-09-18T16:42:53.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-test 2024-09-18T16:42:53.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-test 2024-09-18T16:42:53.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-09-18T16:42:53.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:42:53.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-18T16:42:53.419 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:42:53.419 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:42:53.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:42:53.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:42:53.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:42:53.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:42:53.422 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:42:53.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:42:53.424 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:42:53.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:42:53.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:42:53.427 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:42:53.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:42:53.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:42:53.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:42:53.430 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:42:53.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:42:53.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:42:53.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-09-18T16:42:53.436 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:42:53.436 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:42:53.436 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:42:53.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.562265 2024-09-18T16:42:53.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:42:53.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:42:53.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-test 2024-09-18T16:42:53.441 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T16:42:53.441 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:42:53.441 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:42:53.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.562265 2024-09-18T16:42:53.444 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T16:42:53.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T16:42:53.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T16:42:53.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-09-18T16:42:53.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: TEST_scrub_test td/osd-scrub-test 2024-09-18T16:42:53.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:39: TEST_scrub_test: local dir=td/osd-scrub-test 2024-09-18T16:42:53.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:40: TEST_scrub_test: local poolname=test 2024-09-18T16:42:53.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:41: TEST_scrub_test: local OSDS=3 2024-09-18T16:42:53.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:42: TEST_scrub_test: local objects=15 2024-09-18T16:42:53.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: TEST_scrub_test: TESTDATA=testdata.562265 2024-09-18T16:42:53.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:46: TEST_scrub_test: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2024-09-18T16:42:53.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-test 2024-09-18T16:42:53.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T16:42:53.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T16:42:53.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T16:42:53.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-test/a 2024-09-18T16:42:53.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2024-09-18T16:42:53.840 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T16:42:53.840 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:42:53.840 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:42:53.841 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:42:53.841 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:42:53.841 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:42:53.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:42:53.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-18T16:42:54.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T16:42:54.114 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T16:42:54.115 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:42:54.115 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:42:54.115 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T16:42:54.115 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T16:42:54.116 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:42:54.116 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:42:54.116 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:42:54.117 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:42:54.117 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:42:54.118 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:42:54.118 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.562265/ceph-mon.a.asok 2024-09-18T16:42:54.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:42:54.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.562265/ceph-mon.a.asok config get fsid 2024-09-18T16:42:54.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T16:42:54.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:42:54.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:42:54.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T16:42:54.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T16:42:54.216 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:42:54.216 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:42:54.217 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:42:54.217 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:42:54.217 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:42:54.217 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:42:54.218 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.562265/ceph-mon.a.asok 2024-09-18T16:42:54.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:42:54.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.562265/ceph-mon.a.asok config get mon_host 2024-09-18T16:42:54.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: TEST_scrub_test: run_mgr td/osd-scrub-test x 2024-09-18T16:42:54.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-test 2024-09-18T16:42:54.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T16:42:54.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T16:42:54.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T16:42:54.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-test/x 2024-09-18T16:42:54.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T16:42:55.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T16:42:55.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:42:55.164 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:42:55.165 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:42:55.165 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:42:55.165 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:42:55.165 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:42:55.166 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:42:55.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:42:55.201 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:48: TEST_scrub_test: expr 3 - 1 2024-09-18T16:42:55.201 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:48: TEST_scrub_test: seq 0 2 2024-09-18T16:42:55.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:48: TEST_scrub_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:42:55.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:50: TEST_scrub_test: run_osd td/osd-scrub-test 0 2024-09-18T16:42:55.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-09-18T16:42:55.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:42:55.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T16:42:55.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:42:55.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/0 2024-09-18T16:42:55.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-18T16:42:55.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:42:55.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:42:55.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:42:55.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-09-18T16:42:55.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-09-18T16:42:55.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:42:55.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:42:55.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-18T16:42:55.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:42:55.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:42:55.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:42:55.205 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:42:55.206 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:42:55.206 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:42:55.206 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:42:55.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:42:55.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:42:55.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:42:55.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:42:55.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-18T16:42:55.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-18T16:42:55.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:42:55.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:42:55.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:42:55.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:42:55.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:42:55.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:42:55.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/0 2024-09-18T16:42:55.209 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:42:55.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f5b28faa-549d-445c-9cce-f2be23b1a3c9 2024-09-18T16:42:55.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f5b28faa-549d-445c-9cce-f2be23b1a3c9' 2024-09-18T16:42:55.210 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 f5b28faa-549d-445c-9cce-f2be23b1a3c9 2024-09-18T16:42:55.211 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:42:55.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAPA+tmd8tqDRAAuOGT81+6aWaGrIh05WeyMQ== 2024-09-18T16:42:55.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAPA+tmd8tqDRAAuOGT81+6aWaGrIh05WeyMQ=="}' 2024-09-18T16:42:55.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f5b28faa-549d-445c-9cce-f2be23b1a3c9 -i td/osd-scrub-test/0/new.json 2024-09-18T16:42:55.498 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:42:55.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/0/new.json 2024-09-18T16:42:55.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAPA+tmd8tqDRAAuOGT81+6aWaGrIh05WeyMQ== --osd-uuid f5b28faa-549d-445c-9cce-f2be23b1a3c9 2024-09-18T16:42:55.538 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:42:55.530+0000 7f9981d58540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:42:55.541 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:42:55.534+0000 7f9981d58540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:42:55.541 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:42:55.538+0000 7f9981d58540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:42:55.541 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:42:55.538+0000 7f9981d58540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:42:55.541 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:42:55.538+0000 7f9981d58540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:42:55.542 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:42:55.538+0000 7f9981d58540 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-09-18T16:42:55.614 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:42:55.610+0000 7f9981d58540 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-09-18T16:43:01.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-09-18T16:43:01.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:43:01.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T16:43:01.653 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T16:43:01.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:43:02.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T16:43:02.017 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T16:43:02.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:43:02.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:43:02.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:43:02.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:43:02.045 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:43:02.042+0000 7f322bd1f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:43:02.047 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:43:02.042+0000 7f322bd1f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:43:02.049 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:43:02.046+0000 7f322bd1f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:43:02.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T16:43:02.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:43:02.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:43:02.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:43:02.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:43:02.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:02.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:43:02.311 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:43:02.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:02.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:43:02.587 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:43:02.582+0000 7f322bd1f540 -1 Falling back to public interface 2024-09-18T16:43:02.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:03.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:03.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:03.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:43:03.663 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:43:03.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:03.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:43:03.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:04.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:04.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:04.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:43:04.969 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:43:04.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:04.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:43:05.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:05.285 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:43:05.282+0000 7f322bd1f540 -1 osd.0 0 log_to_monitors true 2024-09-18T16:43:06.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:06.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:06.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:43:06.275 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:43:06.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:06.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:43:06.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:07.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:07.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:07.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:43:07.584 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:43:07.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:07.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:43:07.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:08.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:08.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:08.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:43:08.902 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:43:08.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:08.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:43:09.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:10.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:10.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:10.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:43:10.217 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:43:10.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:10.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:43:10.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:11.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:11.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:11.530 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:43:11.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:43:11.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:11.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:43:11.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:12.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:12.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:12.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:43:12.844 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:43:12.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:12.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:43:13.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:14.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:14.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:14.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:43:14.247 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:43:14.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:14.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:43:14.559 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:15.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:15.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:15.562 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:43:15.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:43:15.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:15.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:43:15.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:16.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:16.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:16.872 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:43:16.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:43:16.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:16.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:43:17.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:18.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:18.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:18.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:43:18.174 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:43:18.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:43:18.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:18.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:19.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:19.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:19.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:43:19.646 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:43:19.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:19.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:43:20.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:21.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:21.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:21.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:43:21.081 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:43:21.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:21.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:43:21.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:22.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:22.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:22.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:43:22.561 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:43:22.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:22.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:43:23.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:24.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:24.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:24.290 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:43:24.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:43:24.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:24.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:43:25.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:26.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:26.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:26.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:43:26.120 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:43:26.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:26.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:43:26.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:27.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:27.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:27.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:43:27.675 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:43:27.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:27.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:43:28.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:29.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:29.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:29.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T16:43:29.054 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T16:43:29.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:29.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:43:29.373 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/662224,v1:127.0.0.1:6803/662224] [v2:127.0.0.1:6804/662224,v1:127.0.0.1:6805/662224] exists,up f5b28faa-549d-445c-9cce-f2be23b1a3c9 2024-09-18T16:43:29.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:43:29.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:43:29.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:43:29.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:48: TEST_scrub_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:43:29.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:50: TEST_scrub_test: run_osd td/osd-scrub-test 1 2024-09-18T16:43:29.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-09-18T16:43:29.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:43:29.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T16:43:29.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:43:29.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/1 2024-09-18T16:43:29.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-18T16:43:29.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:43:29.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:43:29.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:43:29.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-09-18T16:43:29.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-09-18T16:43:29.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:43:29.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:43:29.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-18T16:43:29.376 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:43:29.377 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:43:29.377 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:43:29.378 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:43:29.378 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:43:29.378 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:43:29.378 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:43:29.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:43:29.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:43:29.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:43:29.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:43:29.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-18T16:43:29.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-18T16:43:29.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:43:29.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:43:29.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:43:29.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:43:29.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:43:29.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:43:29.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/1 2024-09-18T16:43:29.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:43:29.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f0823568-f4bd-4f42-b985-fcc11e4f36ef 2024-09-18T16:43:29.381 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 f0823568-f4bd-4f42-b985-fcc11e4f36ef 2024-09-18T16:43:29.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f0823568-f4bd-4f42-b985-fcc11e4f36ef' 2024-09-18T16:43:29.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:43:29.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAxA+tmrmydFxAAA3GywFqvW2OIuz28iZ8RkQ== 2024-09-18T16:43:29.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAxA+tmrmydFxAAA3GywFqvW2OIuz28iZ8RkQ=="}' 2024-09-18T16:43:29.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f0823568-f4bd-4f42-b985-fcc11e4f36ef -i td/osd-scrub-test/1/new.json 2024-09-18T16:43:29.777 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:43:29.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/1/new.json 2024-09-18T16:43:29.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAxA+tmrmydFxAAA3GywFqvW2OIuz28iZ8RkQ== --osd-uuid f0823568-f4bd-4f42-b985-fcc11e4f36ef 2024-09-18T16:43:29.826 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:43:29.822+0000 7f578318c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:43:29.834 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:43:29.830+0000 7f578318c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:43:29.835 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:43:29.830+0000 7f578318c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:43:29.836 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:43:29.830+0000 7f578318c540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:43:29.836 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:43:29.830+0000 7f578318c540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:43:29.836 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:43:29.830+0000 7f578318c540 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-09-18T16:43:29.944 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:43:29.942+0000 7f578318c540 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-09-18T16:43:36.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-09-18T16:43:36.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:43:36.212 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T16:43:36.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T16:43:36.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:43:36.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T16:43:36.596 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T16:43:36.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:43:36.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:43:36.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:43:36.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:43:36.618 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:43:36.614+0000 7fb1b6ed3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:43:36.622 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:43:36.618+0000 7fb1b6ed3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:43:36.624 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:43:36.618+0000 7fb1b6ed3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:43:36.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T16:43:36.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:43:36.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T16:43:36.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:43:36.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:43:36.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:36.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:43:36.895 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:43:36.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:36.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:43:37.171 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:43:37.166+0000 7fb1b6ed3540 -1 Falling back to public interface 2024-09-18T16:43:37.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:38.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:38.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:38.198 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:43:38.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:43:38.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:38.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:43:38.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:39.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:39.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:39.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:43:39.500 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:43:39.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:39.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:43:39.705 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:43:39.702+0000 7fb1b6ed3540 -1 osd.1 0 log_to_monitors true 2024-09-18T16:43:39.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:40.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:40.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:40.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:43:40.815 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:43:40.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:40.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:43:41.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:42.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:42.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:42.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:43:42.120 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:43:42.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:42.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:43:42.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:43.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:43.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:43.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:43:43.490 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:43:43.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:43.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:43:43.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:44.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:44.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:44.832 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:43:44.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:43:44.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:44.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:43:45.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:46.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:46.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:46.141 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:43:46.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:43:46.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:46.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:43:46.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:47.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:47.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:47.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:43:47.470 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:43:47.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:47.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:43:47.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:48.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:48.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:48.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:43:48.781 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:43:48.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:48.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:43:49.097 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:50.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:50.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:50.099 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:43:50.100 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:43:50.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:50.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:43:50.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:51.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:51.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:51.473 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:43:51.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:43:51.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:51.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:43:51.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:52.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:52.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:52.779 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:43:52.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:43:52.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:52.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:43:53.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:54.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:54.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:54.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:43:54.130 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:43:54.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:54.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:43:54.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:55.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:55.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:55.527 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:43:55.527 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:43:55.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:55.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:43:55.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:56.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:56.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:56.883 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:43:56.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:43:56.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:56.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:43:57.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:58.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:58.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:58.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:43:58.240 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:43:58.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:58.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:43:58.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:43:59.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:43:59.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:43:59.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:43:59.760 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:43:59.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:43:59.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:44:00.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:44:01.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:44:01.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:44:01.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:44:01.494 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:44:01.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:44:01.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:44:02.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:44:03.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:44:03.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:44:03.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T16:44:03.222 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T16:44:03.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:44:03.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:44:03.536 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/664028,v1:127.0.0.1:6811/664028] [v2:127.0.0.1:6812/664028,v1:127.0.0.1:6813/664028] exists,up f0823568-f4bd-4f42-b985-fcc11e4f36ef 2024-09-18T16:44:03.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:44:03.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:44:03.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:44:03.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:48: TEST_scrub_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T16:44:03.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:50: TEST_scrub_test: run_osd td/osd-scrub-test 2 2024-09-18T16:44:03.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-09-18T16:44:03.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:44:03.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T16:44:03.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:44:03.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/2 2024-09-18T16:44:03.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-18T16:44:03.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:44:03.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:44:03.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:44:03.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2024-09-18T16:44:03.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2024-09-18T16:44:03.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:44:03.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:44:03.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-18T16:44:03.542 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:44:03.542 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:44:03.542 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:44:03.542 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:44:03.542 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:44:03.542 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:44:03.542 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:44:03.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:44:03.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:44:03.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:44:03.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:44:03.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-18T16:44:03.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-18T16:44:03.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:44:03.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:44:03.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:44:03.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:44:03.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:44:03.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:44:03.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/2 2024-09-18T16:44:03.544 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:44:03.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2a0d1562-860f-4691-8cea-bbd184a58aa0 2024-09-18T16:44:03.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2a0d1562-860f-4691-8cea-bbd184a58aa0' 2024-09-18T16:44:03.545 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 2a0d1562-860f-4691-8cea-bbd184a58aa0 2024-09-18T16:44:03.546 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:44:03.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBTA+tmxjmbIRAAUyJGmwrpb0aaKf7viGV0Fg== 2024-09-18T16:44:03.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBTA+tmxjmbIRAAUyJGmwrpb0aaKf7viGV0Fg=="}' 2024-09-18T16:44:03.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2a0d1562-860f-4691-8cea-bbd184a58aa0 -i td/osd-scrub-test/2/new.json 2024-09-18T16:44:04.091 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:44:04.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/2/new.json 2024-09-18T16:44:04.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBTA+tmxjmbIRAAUyJGmwrpb0aaKf7viGV0Fg== --osd-uuid 2a0d1562-860f-4691-8cea-bbd184a58aa0 2024-09-18T16:44:04.137 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:44:04.134+0000 7f320a3a9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:44:04.140 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:44:04.138+0000 7f320a3a9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:44:04.142 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:44:04.138+0000 7f320a3a9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:44:04.143 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:44:04.138+0000 7f320a3a9540 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-09-18T16:44:04.143 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:44:04.138+0000 7f320a3a9540 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-09-18T16:44:04.143 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:44:04.138+0000 7f320a3a9540 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-09-18T16:44:04.224 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:44:04.218+0000 7f320a3a9540 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-09-18T16:44:10.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2024-09-18T16:44:10.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:44:10.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T16:44:10.270 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T16:44:10.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:44:10.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T16:44:10.653 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T16:44:10.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:44:10.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:44:10.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:44:10.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:44:10.681 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:44:10.678+0000 7f8f579c9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:44:10.684 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:44:10.678+0000 7f8f579c9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:44:10.685 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:44:10.682+0000 7f8f579c9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:44:10.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T16:44:10.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:44:10.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T16:44:10.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:44:10.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:44:10.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:44:10.936 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:44:10.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:44:10.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:44:10.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:44:11.227 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:44:11.222+0000 7f8f579c9540 -1 Falling back to public interface 2024-09-18T16:44:11.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:44:12.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:44:12.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:44:12.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:44:12.278 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:44:12.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:44:12.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:44:12.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:44:13.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:44:13.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:44:13.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:44:13.591 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:44:13.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:44:13.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:44:13.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:44:14.155 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:44:14.150+0000 7f8f579c9540 -1 osd.2 0 log_to_monitors true 2024-09-18T16:44:14.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:44:14.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:44:14.906 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:44:14.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:44:14.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:44:14.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:44:15.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:44:16.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:44:16.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:44:16.247 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:44:16.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:44:16.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:44:16.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:44:16.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:44:17.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:44:17.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:44:17.564 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:44:17.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:44:17.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:44:17.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:44:17.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:44:18.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:44:18.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:44:18.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:44:18.949 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:44:18.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:44:18.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:44:19.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:44:20.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:44:20.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:44:20.259 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:44:20.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:44:20.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:44:20.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:44:20.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:44:21.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:44:21.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:44:21.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:44:21.569 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:44:21.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:44:21.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:44:21.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:44:22.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:44:22.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:44:22.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:44:22.961 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:44:22.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:44:22.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:44:23.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:44:24.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:44:24.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:44:24.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:44:24.272 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:44:24.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:44:24.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:44:24.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:44:25.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:44:25.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:44:25.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:44:25.582 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:44:25.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:44:25.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:44:25.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:44:26.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:44:26.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:44:26.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:44:26.908 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:44:26.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:44:26.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:44:27.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:44:28.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:44:28.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:44:28.566 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:44:28.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:44:28.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:44:28.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:44:28.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:44:29.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:44:29.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:44:29.986 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:44:29.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:44:29.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:44:29.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:44:30.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:44:31.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:44:31.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:44:31.408 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:44:31.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:44:31.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:44:31.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:44:31.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:44:32.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:44:32.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:44:32.841 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:44:32.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:44:32.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:44:32.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:44:33.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:44:34.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:44:34.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:44:34.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:44:34.504 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:44:34.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:44:34.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:44:35.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:44:36.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:44:36.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:44:36.032 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:44:36.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:44:36.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:44:36.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:44:36.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:44:37.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:44:37.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:44:37.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T16:44:37.454 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-09-18T16:44:37.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:44:37.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:44:37.801 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/665832,v1:127.0.0.1:6819/665832] [v2:127.0.0.1:6820/665832,v1:127.0.0.1:6821/665832] exists,up 2a0d1562-860f-4691-8cea-bbd184a58aa0 2024-09-18T16:44:37.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:44:37.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:44:37.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:44:37.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:54: TEST_scrub_test: create_pool test 1 1 2024-09-18T16:44:37.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-18T16:44:38.246 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'test' created 2024-09-18T16:44:38.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T16:44:39.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:55: TEST_scrub_test: wait_for_clean 2024-09-18T16:44:39.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:44:39.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:44:39.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:44:39.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:44:39.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:44:39.262 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:44:39.262 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:44:39.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:44:39.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:44:39.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:44:39.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:44:39.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:44:39.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:44:39.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=500 2024-09-18T16:44:39.485 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:44:39.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:44:39.899 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:44:39.899 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:44:39.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:44:39.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:44:39.900 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:44:40.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836502 2024-09-18T16:44:40.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836502 2024-09-18T16:44:40.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836502' 2024-09-18T16:44:40.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:44:40.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:44:40.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672977 2024-09-18T16:44:40.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672977 2024-09-18T16:44:40.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836502 1-42949672977' 2024-09-18T16:44:40.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:44:40.204 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:44:40.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509452 2024-09-18T16:44:40.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509452 2024-09-18T16:44:40.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836502 1-42949672977 2-64424509452' 2024-09-18T16:44:40.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:44:40.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836502 2024-09-18T16:44:40.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:44:40.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:44:40.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836502 2024-09-18T16:44:40.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:44:40.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836502 2024-09-18T16:44:40.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-18T16:44:40.343 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836502 2024-09-18T16:44:40.343 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:44:40.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836500 -lt 21474836502 2024-09-18T16:44:40.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:44:41.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-18T16:44:41.650 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:44:41.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836502 -lt 21474836502 2024-09-18T16:44:41.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:44:41.960 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672977 2024-09-18T16:44:41.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:44:41.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:44:41.963 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672977 2024-09-18T16:44:41.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:44:41.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672977 2024-09-18T16:44:41.966 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672977 2024-09-18T16:44:41.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-18T16:44:41.967 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:44:42.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-18T16:44:42.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:44:42.310 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509452 2024-09-18T16:44:42.310 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:44:42.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:44:42.313 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509452 2024-09-18T16:44:42.313 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:44:42.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509452 2024-09-18T16:44:42.315 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-18T16:44:42.315 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509452 2024-09-18T16:44:42.315 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:44:42.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509452 -lt 64424509452 2024-09-18T16:44:42.664 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:44:42.665 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:44:42.665 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:44:43.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:44:43.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:44:43.065 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:44:43.065 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:44:43.065 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:44:43.065 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:44:43.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:44:43.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:44:43.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:44:43.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:44:43.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:44:43.406 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:44:43.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:44:43.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:44:43.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:44:43.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:56: TEST_scrub_test: ceph osd dump 2024-09-18T16:44:43.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:56: TEST_scrub_test: grep '^pool.*['\'']test['\'']' 2024-09-18T16:44:43.758 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:56: TEST_scrub_test: awk '{ print $2 }' 2024-09-18T16:44:44.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:56: TEST_scrub_test: poolid=1 2024-09-18T16:44:44.101 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:58: TEST_scrub_test: dd if=/dev/urandom of=testdata.562265 bs=1032 count=1 2024-09-18T16:44:44.102 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records in 2024-09-18T16:44:44.102 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records out 2024-09-18T16:44:44.102 INFO:tasks.workunit.client.0.smithi046.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 0.000175518 s, 5.9 MB/s 2024-09-18T16:44:44.103 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:59: TEST_scrub_test: seq 1 15 2024-09-18T16:44:44.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:59: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-18T16:44:44.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:61: TEST_scrub_test: rados -p test put obj1 testdata.562265 2024-09-18T16:44:44.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:59: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-18T16:44:44.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:61: TEST_scrub_test: rados -p test put obj2 testdata.562265 2024-09-18T16:44:44.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:59: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-18T16:44:44.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:61: TEST_scrub_test: rados -p test put obj3 testdata.562265 2024-09-18T16:44:44.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:59: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-18T16:44:44.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:61: TEST_scrub_test: rados -p test put obj4 testdata.562265 2024-09-18T16:44:44.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:59: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-18T16:44:44.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:61: TEST_scrub_test: rados -p test put obj5 testdata.562265 2024-09-18T16:44:44.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:59: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-18T16:44:44.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:61: TEST_scrub_test: rados -p test put obj6 testdata.562265 2024-09-18T16:44:45.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:59: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-18T16:44:45.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:61: TEST_scrub_test: rados -p test put obj7 testdata.562265 2024-09-18T16:44:45.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:59: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-18T16:44:45.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:61: TEST_scrub_test: rados -p test put obj8 testdata.562265 2024-09-18T16:44:45.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:59: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-18T16:44:45.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:61: TEST_scrub_test: rados -p test put obj9 testdata.562265 2024-09-18T16:44:45.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:59: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-18T16:44:45.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:61: TEST_scrub_test: rados -p test put obj10 testdata.562265 2024-09-18T16:44:45.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:59: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-18T16:44:45.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:61: TEST_scrub_test: rados -p test put obj11 testdata.562265 2024-09-18T16:44:45.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:59: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-18T16:44:45.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:61: TEST_scrub_test: rados -p test put obj12 testdata.562265 2024-09-18T16:44:45.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:59: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-18T16:44:45.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:61: TEST_scrub_test: rados -p test put obj13 testdata.562265 2024-09-18T16:44:46.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:59: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-18T16:44:46.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:61: TEST_scrub_test: rados -p test put obj14 testdata.562265 2024-09-18T16:44:46.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:59: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-18T16:44:46.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:61: TEST_scrub_test: rados -p test put obj15 testdata.562265 2024-09-18T16:44:46.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:63: TEST_scrub_test: rm -f testdata.562265 2024-09-18T16:44:46.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:65: TEST_scrub_test: get_primary test obj1 2024-09-18T16:44:46.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T16:44:46.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T16:44:46.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T16:44:46.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T16:44:46.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:65: TEST_scrub_test: local primary=1 2024-09-18T16:44:46.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:66: TEST_scrub_test: get_not_primary test obj1 2024-09-18T16:44:46.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-09-18T16:44:46.707 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-09-18T16:44:46.707 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-09-18T16:44:46.708 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T16:44:46.708 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T16:44:46.708 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T16:44:46.708 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T16:44:47.025 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T16:44:47.025 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-09-18T16:44:47.025 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T16:44:47.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:66: TEST_scrub_test: local otherosd=0 2024-09-18T16:44:47.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:67: TEST_scrub_test: '[' 0 = 2 ']' 2024-09-18T16:44:47.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:71: TEST_scrub_test: local anotherosd=2 2024-09-18T16:44:47.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:74: TEST_scrub_test: objectstore_tool td/osd-scrub-test 2 obj1 set-bytes /etc/fstab 2024-09-18T16:44:47.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-test 2024-09-18T16:44:47.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T16:44:47.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-18T16:44:47.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T16:44:47.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-test 2 obj1 set-bytes /etc/fstab 2024-09-18T16:44:47.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-test 2024-09-18T16:44:47.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T16:44:47.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-18T16:44:47.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T16:44:47.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-test TERM osd.2 2024-09-18T16:44:47.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:44:47.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:44:47.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:44:47.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:44:47.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:44:47.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:44:47.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-test 2 obj1 set-bytes /etc/fstab 2024-09-18T16:44:47.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-test 2024-09-18T16:44:47.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T16:44:47.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-18T16:44:47.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T16:44:47.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-test/2 2024-09-18T16:44:47.720 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-test/2 obj1 set-bytes /etc/fstab 2024-09-18T16:44:49.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-test 2 2024-09-18T16:44:49.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-test 2024-09-18T16:44:49.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T16:44:49.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T16:44:49.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T16:44:49.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-test/2 2024-09-18T16:44:49.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-18T16:44:49.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:44:49.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:44:49.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:44:49.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2024-09-18T16:44:49.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2024-09-18T16:44:49.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T16:44:49.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T16:44:49.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-18T16:44:49.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T16:44:49.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:44:49.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:44:49.983 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:44:49.983 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:44:49.983 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:44:49.983 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:44:49.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' 2024-09-18T16:44:49.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:44:49.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-18T16:44:49.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-18T16:44:49.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:44:49.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:44:49.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:44:49.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:44:49.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T16:44:49.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T16:44:49.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-test/2 2024-09-18T16:44:49.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T16:44:49.985 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.2 2024-09-18T16:44:49.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=cede4c3f-f7be-455a-b099-33fa1340bd23 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.562265/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:44:49.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-test/2/whoami 2024-09-18T16:44:49.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T16:44:49.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T16:44:49.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T16:44:49.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T16:44:50.017 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:44:50.013+0000 7fac3f9dc540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:44:50.019 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:44:50.013+0000 7fac3f9dc540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:44:50.021 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:44:50.017+0000 7fac3f9dc540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:44:50.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T16:44:50.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:44:50.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T16:44:50.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:44:50.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:44:50.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:44:50.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:44:50.283 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T16:44:50.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:44:50.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:44:50.571 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:44:50.565+0000 7fac3f9dc540 -1 Falling back to public interface 2024-09-18T16:44:50.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:44:51.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:44:51.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:44:51.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:44:51.626 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:44:51.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:44:51.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:44:51.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:44:52.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:44:52.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:44:52.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:44:52.936 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T16:44:52.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:44:52.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:44:53.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:44:53.326 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:44:53.321+0000 7fac3f9dc540 -1 osd.2 19 log_to_monitors true 2024-09-18T16:44:54.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:44:54.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-09-18T16:44:54.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:44:54.246 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T16:44:54.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:44:54.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:44:54.612 INFO:tasks.workunit.client.0.smithi046.stderr:osd.2 up in weight 1 up_from 23 up_thru 0 down_at 20 last_clean_interval [15,19) [v2:127.0.0.1:6818/668016,v1:127.0.0.1:6819/668016] [v2:127.0.0.1:6820/668016,v1:127.0.0.1:6821/668016] exists,up 2a0d1562-860f-4691-8cea-bbd184a58aa0 2024-09-18T16:44:54.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:44:54.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:44:54.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:44:54.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T16:44:54.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:44:54.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:44:54.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:44:54.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:44:54.614 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:44:54.615 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:44:54.615 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:44:54.615 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:44:54.615 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:44:54.615 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:44:54.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:44:54.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:44:54.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:44:54.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=500 2024-09-18T16:44:54.826 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:44:55.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:44:55.126 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:44:55.126 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:44:55.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:44:55.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:44:55.127 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:44:55.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836506 2024-09-18T16:44:55.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836506 2024-09-18T16:44:55.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836506' 2024-09-18T16:44:55.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:44:55.300 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:44:55.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672981 2024-09-18T16:44:55.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672981 2024-09-18T16:44:55.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836506 1-42949672981' 2024-09-18T16:44:55.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:44:55.454 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:44:55.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=98784247810 2024-09-18T16:44:55.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 98784247810 2024-09-18T16:44:55.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836506 1-42949672981 2-98784247810' 2024-09-18T16:44:55.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:44:55.585 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836506 2024-09-18T16:44:55.586 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:44:55.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:44:55.589 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836506 2024-09-18T16:44:55.589 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:44:55.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836506 2024-09-18T16:44:55.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-18T16:44:55.592 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836506 2024-09-18T16:44:55.592 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:44:55.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836505 -lt 21474836506 2024-09-18T16:44:55.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:44:56.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-18T16:44:56.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:44:57.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836506 -lt 21474836506 2024-09-18T16:44:57.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:44:57.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672981 2024-09-18T16:44:57.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:44:57.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:44:57.210 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672981 2024-09-18T16:44:57.210 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:44:57.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672981 2024-09-18T16:44:57.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-09-18T16:44:57.212 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 42949672981 2024-09-18T16:44:57.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:44:57.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672981 -lt 42949672981 2024-09-18T16:44:57.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:44:57.562 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-98784247810 2024-09-18T16:44:57.563 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:44:57.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:44:57.566 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-98784247810 2024-09-18T16:44:57.566 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:44:57.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=98784247810 2024-09-18T16:44:57.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 98784247810' 2024-09-18T16:44:57.568 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 98784247810 2024-09-18T16:44:57.569 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:44:57.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 98784247810 -lt 98784247810 2024-09-18T16:44:57.912 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:44:57.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:44:57.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:44:58.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:44:58.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:44:58.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:44:58.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:44:58.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:44:58.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:44:58.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:44:58.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:44:58.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:44:58.649 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:44:58.650 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:44:58.650 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:44:59.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:44:59.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:44:59.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:44:59.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:76: TEST_scrub_test: local pgid=1.0 2024-09-18T16:44:59.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:77: TEST_scrub_test: pg_deep_scrub 1.0 2024-09-18T16:44:59.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: pg_deep_scrub: local pgid=1.0 2024-09-18T16:44:59.005 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: pg_deep_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2024-09-18T16:44:59.005 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:44:59.005 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T16:44:59.006 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:44:59.006 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2024-09-18T16:44:59.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: pg_deep_scrub: local last_scrub=2024-09-18T16:44:38.155614+0000 2024-09-18T16:44:59.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: pg_deep_scrub: ceph pg deep-scrub 1.0 2024-09-18T16:44:59.598 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 1.0 on osd.1 to deep-scrub 2024-09-18T16:44:59.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: pg_deep_scrub: wait_for_scrub 1.0 2024-09-18T16:44:38.155614+0000 last_deep_scrub_stamp 2024-09-18T16:44:59.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-18T16:44:59.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T16:44:38.155614+0000 2024-09-18T16:44:59.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-09-18T16:44:59.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T16:44:59.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:44:59.609 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2024-09-18T16:44:59.610 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:44:59.610 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T16:44:59.610 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:44:59.610 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2024-09-18T16:44:59.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:44:38.155614+0000 '>' 2024-09-18T16:44:38.155614+0000 2024-09-18T16:44:59.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:45:00.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:45:00.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:45:00.906 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2024-09-18T16:45:00.906 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:45:00.906 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T16:45:00.907 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:45:00.907 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2024-09-18T16:45:01.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:44:38.155614+0000 '>' 2024-09-18T16:44:38.155614+0000 2024-09-18T16:45:01.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:45:02.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:45:02.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:45:02.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2024-09-18T16:45:02.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:45:02.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T16:45:02.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:45:02.219 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2024-09-18T16:45:02.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:44:38.155614+0000 '>' 2024-09-18T16:44:38.155614+0000 2024-09-18T16:45:02.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:45:03.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:45:03.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:45:03.538 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2024-09-18T16:45:03.538 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:45:03.538 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T16:45:03.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:45:03.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2024-09-18T16:45:03.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:44:38.155614+0000 '>' 2024-09-18T16:44:38.155614+0000 2024-09-18T16:45:03.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:45:04.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:45:04.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:45:04.845 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2024-09-18T16:45:04.845 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:45:04.845 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T16:45:04.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:45:04.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2024-09-18T16:45:05.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:45:00.032332+0000 '>' 2024-09-18T16:44:38.155614+0000 2024-09-18T16:45:05.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T16:45:05.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:79: TEST_scrub_test: ceph pg dump pgs 2024-09-18T16:45:05.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:79: TEST_scrub_test: grep -q -- +inconsistent 2024-09-18T16:45:05.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:79: TEST_scrub_test: grep '^1.0' 2024-09-18T16:45:05.445 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:45:05.456 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:80: TEST_scrub_test: ceph pg 1.0 query 2024-09-18T16:45:05.456 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:80: TEST_scrub_test: jq .info.stats.stat_sum.num_scrub_errors 2024-09-18T16:45:05.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:80: TEST_scrub_test: test 2 = 2 2024-09-18T16:45:05.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:82: TEST_scrub_test: ceph osd out 1 2024-09-18T16:45:06.018 INFO:tasks.workunit.client.0.smithi046.stderr:marked out osd.1. 2024-09-18T16:45:06.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:83: TEST_scrub_test: wait_for_clean 2024-09-18T16:45:06.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:45:06.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:45:06.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:45:06.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:45:06.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:45:06.032 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:45:06.032 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:45:06.033 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:45:06.033 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:45:06.033 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:45:06.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:45:06.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:45:06.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:45:06.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=500 2024-09-18T16:45:06.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:45:06.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:45:06.632 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:45:06.632 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:45:06.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:45:06.632 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:45:06.633 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:45:06.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836509 2024-09-18T16:45:06.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836509 2024-09-18T16:45:06.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509' 2024-09-18T16:45:06.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:45:06.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:45:07.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672984 2024-09-18T16:45:07.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672984 2024-09-18T16:45:07.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509 1-42949672984' 2024-09-18T16:45:07.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:45:07.006 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:45:07.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=98784247813 2024-09-18T16:45:07.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 98784247813 2024-09-18T16:45:07.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509 1-42949672984 2-98784247813' 2024-09-18T16:45:07.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:45:07.137 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:45:07.137 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836509 2024-09-18T16:45:07.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:45:07.139 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836509 2024-09-18T16:45:07.140 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:45:07.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836509 2024-09-18T16:45:07.142 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836509 2024-09-18T16:45:07.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-09-18T16:45:07.142 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:45:07.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836508 -lt 21474836509 2024-09-18T16:45:07.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:45:08.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-18T16:45:08.569 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:45:08.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836509 -lt 21474836509 2024-09-18T16:45:08.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:45:08.887 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672984 2024-09-18T16:45:08.887 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:45:08.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:45:08.890 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672984 2024-09-18T16:45:08.891 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:45:08.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672984 2024-09-18T16:45:08.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-09-18T16:45:08.892 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672984 2024-09-18T16:45:08.893 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:45:09.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672985 -lt 42949672984 2024-09-18T16:45:09.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:45:09.239 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-98784247813 2024-09-18T16:45:09.239 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:45:09.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:45:09.243 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-98784247813 2024-09-18T16:45:09.243 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:45:09.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=98784247813 2024-09-18T16:45:09.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 98784247813' 2024-09-18T16:45:09.244 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 98784247813 2024-09-18T16:45:09.245 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:45:09.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 98784247813 -lt 98784247813 2024-09-18T16:45:09.586 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:45:09.587 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:45:09.587 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:45:09.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:45:09.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:45:09.944 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:45:09.944 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:45:09.944 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:45:09.944 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:45:09.945 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:45:09.945 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:45:10.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T16:45:10.299 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:45:10.299 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:45:10.300 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:45:10.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T16:45:10.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-18T16:45:10.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T16:45:10.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-18T16:45:10.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T16:45:10.661 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T16:45:10.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T16:45:10.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:45:10.763 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:45:10.763 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:45:10.763 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:45:10.763 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:45:10.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:45:10.764 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:45:11.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:45:11.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:45:11.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:45:11.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:45:11.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:45:11.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:45:11.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:45:11.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:85: TEST_scrub_test: pg_deep_scrub 1.0 2024-09-18T16:45:11.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: pg_deep_scrub: local pgid=1.0 2024-09-18T16:45:11.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: pg_deep_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2024-09-18T16:45:11.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:45:11.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T16:45:11.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:45:11.409 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2024-09-18T16:45:11.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: pg_deep_scrub: local last_scrub=2024-09-18T16:45:00.032332+0000 2024-09-18T16:45:11.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: pg_deep_scrub: ceph pg deep-scrub 1.0 2024-09-18T16:45:12.006 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 1.0 on osd.0 to deep-scrub 2024-09-18T16:45:12.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: pg_deep_scrub: wait_for_scrub 1.0 2024-09-18T16:45:00.032332+0000 last_deep_scrub_stamp 2024-09-18T16:45:12.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-18T16:45:12.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T16:45:00.032332+0000 2024-09-18T16:45:12.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-09-18T16:45:12.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T16:45:12.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:45:12.020 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2024-09-18T16:45:12.020 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:45:12.020 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T16:45:12.020 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:45:12.020 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2024-09-18T16:45:12.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:45:00.032332+0000 '>' 2024-09-18T16:45:00.032332+0000 2024-09-18T16:45:12.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:45:13.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:45:13.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:45:13.318 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2024-09-18T16:45:13.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:45:13.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T16:45:13.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:45:13.320 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2024-09-18T16:45:13.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:45:00.032332+0000 '>' 2024-09-18T16:45:00.032332+0000 2024-09-18T16:45:13.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:45:14.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:45:14.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:45:14.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2024-09-18T16:45:14.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:45:14.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-18T16:45:14.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:45:14.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2024-09-18T16:45:14.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:45:12.809544+0000 '>' 2024-09-18T16:45:00.032332+0000 2024-09-18T16:45:14.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T16:45:14.942 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:87: TEST_scrub_test: ceph pg 1.0 query 2024-09-18T16:45:14.942 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:87: TEST_scrub_test: jq .info.stats.stat_sum.num_scrub_errors 2024-09-18T16:45:15.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:87: TEST_scrub_test: test 2 = 2 2024-09-18T16:45:15.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:88: TEST_scrub_test: ceph pg 1.0 query 2024-09-18T16:45:15.075 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:88: TEST_scrub_test: jq '.peer_info[0].stats.stat_sum.num_scrub_errors' 2024-09-18T16:45:15.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:88: TEST_scrub_test: test 2 = 2 2024-09-18T16:45:15.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:89: TEST_scrub_test: ceph pg dump pgs 2024-09-18T16:45:15.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:89: TEST_scrub_test: grep '^1.0' 2024-09-18T16:45:15.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:89: TEST_scrub_test: grep -q -- +inconsistent 2024-09-18T16:45:15.500 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:45:15.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:91: TEST_scrub_test: ceph osd in 1 2024-09-18T16:45:15.946 INFO:tasks.workunit.client.0.smithi046.stderr:marked in osd.1. 2024-09-18T16:45:15.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:92: TEST_scrub_test: wait_for_clean 2024-09-18T16:45:15.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:45:15.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:45:15.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:45:15.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:45:15.958 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:45:15.959 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:45:15.959 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:45:15.960 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:45:15.960 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:45:15.960 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:45:16.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:45:16.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:45:16.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:45:16.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=500 2024-09-18T16:45:16.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:45:16.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:45:16.563 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:45:16.563 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:45:16.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:45:16.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:45:16.563 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:45:16.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836512 2024-09-18T16:45:16.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836512 2024-09-18T16:45:16.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836512' 2024-09-18T16:45:16.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:45:16.696 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:45:16.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672987 2024-09-18T16:45:16.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672987 2024-09-18T16:45:16.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836512 1-42949672987' 2024-09-18T16:45:16.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:45:16.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:45:17.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=98784247816 2024-09-18T16:45:17.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 98784247816 2024-09-18T16:45:17.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836512 1-42949672987 2-98784247816' 2024-09-18T16:45:17.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:45:17.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836512 2024-09-18T16:45:17.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:45:17.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:45:17.055 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836512 2024-09-18T16:45:17.056 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:45:17.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836512 2024-09-18T16:45:17.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-09-18T16:45:17.058 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836512 2024-09-18T16:45:17.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:45:17.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836511 -lt 21474836512 2024-09-18T16:45:17.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:45:18.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-18T16:45:18.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:45:18.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836511 -lt 21474836512 2024-09-18T16:45:18.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:45:19.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 499 -eq 0 ']' 2024-09-18T16:45:19.668 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:45:19.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836512 -lt 21474836512 2024-09-18T16:45:19.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:45:19.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672987 2024-09-18T16:45:19.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:45:19.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:45:19.984 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672987 2024-09-18T16:45:19.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:45:19.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672987 2024-09-18T16:45:19.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-09-18T16:45:19.987 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672987 2024-09-18T16:45:19.987 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:45:20.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672988 -lt 42949672987 2024-09-18T16:45:20.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:45:20.329 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-98784247816 2024-09-18T16:45:20.329 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:45:20.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:45:20.330 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-98784247816 2024-09-18T16:45:20.330 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:45:20.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=98784247816 2024-09-18T16:45:20.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 98784247816' 2024-09-18T16:45:20.332 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 98784247816 2024-09-18T16:45:20.332 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:45:20.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 98784247816 -lt 98784247816 2024-09-18T16:45:20.671 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:45:20.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:45:20.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:45:21.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:45:21.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:45:21.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:45:21.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:45:21.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:45:21.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:45:21.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:45:21.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:45:21.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:45:21.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:45:21.425 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:45:21.426 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:45:21.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:45:21.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:45:21.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:45:21.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:94: TEST_scrub_test: repair 1.0 2024-09-18T16:45:21.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=1.0 2024-09-18T16:45:21.789 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 1.0 2024-09-18T16:45:21.789 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:45:21.789 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:45:21.789 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:45:21.789 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:45:22.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-09-18T16:45:12.809544+0000 2024-09-18T16:45:22.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 1.0 2024-09-18T16:45:22.375 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 1.0 on osd.1 to repair 2024-09-18T16:45:22.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 1.0 2024-09-18T16:45:12.809544+0000 2024-09-18T16:45:22.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: wait_for_scrub: local pgid=1.0 2024-09-18T16:45:22.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: wait_for_scrub: local last_scrub=2024-09-18T16:45:12.809544+0000 2024-09-18T16:45:22.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: wait_for_scrub: local sname=last_scrub_stamp 2024-09-18T16:45:22.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i=0 )) 2024-09-18T16:45:22.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:45:22.385 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:45:22.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:45:22.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:45:22.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:45:22.386 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:45:22.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:45:12.809544+0000 '>' 2024-09-18T16:45:12.809544+0000 2024-09-18T16:45:22.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:45:23.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:45:23.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:45:23.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:45:23.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:45:23.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:45:23.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:45:23.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:45:23.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:45:12.809544+0000 '>' 2024-09-18T16:45:12.809544+0000 2024-09-18T16:45:23.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:45:24.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:45:24.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:45:24.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:45:24.986 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:45:24.986 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:45:24.986 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:45:24.986 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:45:25.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:45:12.809544+0000 '>' 2024-09-18T16:45:12.809544+0000 2024-09-18T16:45:25.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:45:26.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:45:26.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:45:26.298 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:45:26.299 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:45:26.299 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:45:26.299 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:45:26.299 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:45:26.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:45:12.809544+0000 '>' 2024-09-18T16:45:12.809544+0000 2024-09-18T16:45:26.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:45:27.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:45:27.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:45:27.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:45:27.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:45:27.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:45:27.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:45:27.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:45:27.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:45:12.809544+0000 '>' 2024-09-18T16:45:12.809544+0000 2024-09-18T16:45:27.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: wait_for_scrub: sleep 1 2024-09-18T16:45:28.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i++ )) 2024-09-18T16:45:28.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: wait_for_scrub: (( i < 500 )) 2024-09-18T16:45:28.963 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-18T16:45:28.963 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-18T16:45:28.963 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-18T16:45:28.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-18T16:45:28.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-18T16:45:29.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: wait_for_scrub: test 2024-09-18T16:45:26.470912+0000 '>' 2024-09-18T16:45:12.809544+0000 2024-09-18T16:45:29.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: wait_for_scrub: return 0 2024-09-18T16:45:29.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:95: TEST_scrub_test: wait_for_clean 2024-09-18T16:45:29.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:45:29.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:45:29.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:45:29.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:45:29.274 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:45:29.275 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:45:29.275 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:45:29.276 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:45:29.276 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:45:29.276 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:45:29.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:45:29.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:45:29.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:45:29.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=500 2024-09-18T16:45:29.498 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:45:29.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:45:29.809 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:45:29.809 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:45:29.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:45:29.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:45:29.810 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:45:29.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836516 2024-09-18T16:45:29.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836516 2024-09-18T16:45:29.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836516' 2024-09-18T16:45:29.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:45:29.939 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:45:30.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672991 2024-09-18T16:45:30.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672991 2024-09-18T16:45:30.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836516 1-42949672991' 2024-09-18T16:45:30.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:45:30.078 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:45:30.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=98784247820 2024-09-18T16:45:30.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 98784247820 2024-09-18T16:45:30.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836516 1-42949672991 2-98784247820' 2024-09-18T16:45:30.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:45:30.219 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836516 2024-09-18T16:45:30.219 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:45:30.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:45:30.221 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836516 2024-09-18T16:45:30.222 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:45:30.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836516 2024-09-18T16:45:30.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-09-18T16:45:30.224 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836516 2024-09-18T16:45:30.224 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:45:30.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836514 -lt 21474836516 2024-09-18T16:45:30.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:45:31.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-18T16:45:31.523 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:45:31.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836516 -lt 21474836516 2024-09-18T16:45:31.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:45:31.830 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672991 2024-09-18T16:45:31.830 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:45:31.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:45:31.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672991 2024-09-18T16:45:31.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:45:31.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672991 2024-09-18T16:45:31.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672991' 2024-09-18T16:45:31.835 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672991 2024-09-18T16:45:31.836 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:45:32.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672991 -lt 42949672991 2024-09-18T16:45:32.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:45:32.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-98784247820 2024-09-18T16:45:32.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:45:32.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:45:32.179 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-98784247820 2024-09-18T16:45:32.179 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:45:32.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=98784247820 2024-09-18T16:45:32.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 98784247820' 2024-09-18T16:45:32.181 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 98784247820 2024-09-18T16:45:32.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:45:32.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 98784247820 -lt 98784247820 2024-09-18T16:45:32.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:45:32.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:45:32.520 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:45:32.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:45:32.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:45:32.879 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:45:32.879 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:45:32.879 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:45:32.879 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:45:32.880 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:45:32.880 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:45:33.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:45:33.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:45:33.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:45:33.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:45:33.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:45:33.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:45:33.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:45:33.589 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:98: TEST_scrub_test: ceph pg 1.0 query 2024-09-18T16:45:33.589 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:98: TEST_scrub_test: jq '.peer_info[0].stats.stat_sum.num_scrub_errors' 2024-09-18T16:45:33.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:98: TEST_scrub_test: test 2 = 2 2024-09-18T16:45:33.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:99: TEST_scrub_test: ceph pg dump pgs 2024-09-18T16:45:33.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:99: TEST_scrub_test: grep '^1.0' 2024-09-18T16:45:33.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:99: TEST_scrub_test: grep -vq -- +inconsistent 2024-09-18T16:45:34.068 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:45:34.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:101: TEST_scrub_test: ceph osd out 1 2024-09-18T16:45:34.622 INFO:tasks.workunit.client.0.smithi046.stderr:marked out osd.1. 2024-09-18T16:45:34.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:102: TEST_scrub_test: wait_for_clean 2024-09-18T16:45:34.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:45:34.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:45:34.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:45:34.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:45:34.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:45:34.637 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:45:34.637 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:45:34.637 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:45:34.637 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:45:34.637 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:45:34.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:45:34.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:45:34.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:45:34.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=500 2024-09-18T16:45:34.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:45:35.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:45:35.302 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:45:35.302 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:45:35.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:45:35.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:45:35.303 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:45:35.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836518 2024-09-18T16:45:35.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836518 2024-09-18T16:45:35.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836518' 2024-09-18T16:45:35.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:45:35.532 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:45:35.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672993 2024-09-18T16:45:35.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672993 2024-09-18T16:45:35.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836518 1-42949672993' 2024-09-18T16:45:35.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:45:35.791 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:45:36.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=98784247822 2024-09-18T16:45:36.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 98784247822 2024-09-18T16:45:36.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836518 1-42949672993 2-98784247822' 2024-09-18T16:45:36.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:45:36.047 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836518 2024-09-18T16:45:36.047 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:45:36.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:45:36.051 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836518 2024-09-18T16:45:36.051 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:45:36.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836518 2024-09-18T16:45:36.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2024-09-18T16:45:36.052 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836518 2024-09-18T16:45:36.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:45:36.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836517 -lt 21474836518 2024-09-18T16:45:36.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:45:37.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-18T16:45:37.369 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:45:37.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836518 -lt 21474836518 2024-09-18T16:45:37.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:45:37.677 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672993 2024-09-18T16:45:37.678 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:45:37.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:45:37.681 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672993 2024-09-18T16:45:37.681 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:45:37.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672993 2024-09-18T16:45:37.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672993' 2024-09-18T16:45:37.683 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672993 2024-09-18T16:45:37.683 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:45:38.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672993 -lt 42949672993 2024-09-18T16:45:38.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:45:38.028 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-98784247822 2024-09-18T16:45:38.028 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:45:38.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:45:38.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-98784247822 2024-09-18T16:45:38.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:45:38.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=98784247822 2024-09-18T16:45:38.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 98784247822' 2024-09-18T16:45:38.034 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 98784247822 2024-09-18T16:45:38.034 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:45:38.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 98784247822 -lt 98784247822 2024-09-18T16:45:38.377 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:45:38.378 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:45:38.378 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:45:38.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:45:38.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:45:38.790 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:45:38.790 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:45:38.790 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:45:38.790 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:45:38.791 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:45:38.791 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:45:39.099 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T16:45:39.100 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:45:39.101 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:45:39.101 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:45:39.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T16:45:39.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-18T16:45:39.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T16:45:39.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-18T16:45:39.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T16:45:39.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T16:45:39.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T16:45:39.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:45:39.554 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:45:39.554 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:45:39.554 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:45:39.555 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:45:39.555 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:45:39.555 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:45:39.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T16:45:39.853 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:45:39.854 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:45:39.854 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:45:40.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T16:45:40.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T16:45:40.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T16:45:40.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T16:45:40.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T16:45:40.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T16:45:40.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T16:45:40.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T16:45:40.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T16:45:40.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T16:45:40.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T16:45:40.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-09-18T16:45:40.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T16:45:40.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-18T16:45:40.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T16:45:40.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:45:40.829 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:45:40.829 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:45:40.829 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:45:40.829 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:45:40.829 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:45:40.830 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:45:41.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:45:41.135 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:45:41.135 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:45:41.135 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:45:41.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:45:41.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:45:41.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:45:41.488 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:104: TEST_scrub_test: ceph pg 1.0 query 2024-09-18T16:45:41.488 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:104: TEST_scrub_test: jq .info.stats.stat_sum.num_scrub_errors 2024-09-18T16:45:41.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:104: TEST_scrub_test: test 0 = 0 2024-09-18T16:45:41.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:105: TEST_scrub_test: ceph pg 1.0 query 2024-09-18T16:45:41.673 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:105: TEST_scrub_test: jq '.peer_info[0].stats.stat_sum.num_scrub_errors' 2024-09-18T16:45:41.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:105: TEST_scrub_test: test 0 = 0 2024-09-18T16:45:41.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:106: TEST_scrub_test: ceph pg 1.0 query 2024-09-18T16:45:41.806 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:106: TEST_scrub_test: jq '.peer_info[1].stats.stat_sum.num_scrub_errors' 2024-09-18T16:45:41.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:106: TEST_scrub_test: test 0 = 0 2024-09-18T16:45:41.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:107: TEST_scrub_test: ceph pg dump pgs 2024-09-18T16:45:41.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:107: TEST_scrub_test: grep '^1.0' 2024-09-18T16:45:41.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:107: TEST_scrub_test: grep -vq -- +inconsistent 2024-09-18T16:45:42.221 INFO:tasks.workunit.client.0.smithi046.stderr:dumped pgs 2024-09-18T16:45:42.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: teardown td/osd-scrub-test 2024-09-18T16:45:42.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-09-18T16:45:42.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:45:42.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-18T16:45:42.231 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:45:42.231 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:45:42.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:45:42.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:45:42.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:45:42.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:45:42.350 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:45:42.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:45:42.352 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:45:42.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:45:42.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:45:42.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:45:42.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:45:42.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:45:42.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:45:42.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:45:42.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:45:42.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:45:42.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-09-18T16:45:42.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:45:42.384 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:45:42.384 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:45:42.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.562265 2024-09-18T16:45:42.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:45:42.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:45:42.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T16:45:42.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-scrub-test 0 2024-09-18T16:45:42.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-09-18T16:45:42.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T16:45:42.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-18T16:45:42.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:45:42.387 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:45:42.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:45:42.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:45:42.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:45:42.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:45:42.392 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:45:42.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:45:42.395 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:45:42.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:45:42.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:45:42.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:45:42.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:45:42.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:45:42.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:45:42.400 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:45:42.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:45:42.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T16:45:42.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-09-18T16:45:42.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:45:42.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:45:42.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.562265 2024-09-18T16:45:42.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.562265 2024-09-18T16:45:42.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:45:42.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:45:42.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T16:45:42.409 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T16:45:42.409 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T16:45:42.423 INFO:tasks.workunit:Running workunit scrub/osd-unexpected-clone.sh... 2024-09-18T16:45:42.423 DEBUG:teuthology.orchestra.run.smithi046:workunit test scrub/osd-unexpected-clone.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh 2024-09-18T16:45:42.441 INFO:tasks.workunit.client.0.smithi046.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T16:45:42.452 INFO:tasks.workunit.client.0.smithi046.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T16:45:42.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-18T16:45:42.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-18T16:45:42.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T16:45:42.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T16:45:42.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T16:45:42.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T16:45:42.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T16:45:42.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T16:45:42.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-unexpected-clone 2024-09-18T16:45:42.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:22: run: local dir=td/osd-unexpected-clone 2024-09-18T16:45:42.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:23: run: shift 2024-09-18T16:45:42.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:25: run: export CEPH_MON=127.0.0.1:7144 2024-09-18T16:45:42.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:25: run: CEPH_MON=127.0.0.1:7144 2024-09-18T16:45:42.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:26: run: export CEPH_ARGS 2024-09-18T16:45:42.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:27: run: uuidgen 2024-09-18T16:45:42.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:27: run: CEPH_ARGS+='--fsid=3fdc75b7-dab6-4baf-bc58-f78ba81ca67a --auth-supported=none ' 2024-09-18T16:45:42.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2024-09-18T16:45:42.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:30: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-09-18T16:45:42.454 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:31: run: set 2024-09-18T16:45:42.454 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T16:45:42.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:31: run: local funcs=TEST_recover_unexpected 2024-09-18T16:45:42.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:32: run: for func in $funcs 2024-09-18T16:45:42.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:33: run: setup td/osd-unexpected-clone 2024-09-18T16:45:42.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-unexpected-clone 2024-09-18T16:45:42.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-unexpected-clone 2024-09-18T16:45:42.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-unexpected-clone 2024-09-18T16:45:42.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:45:42.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-unexpected-clone KILL 2024-09-18T16:45:42.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:45:42.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:45:42.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:45:42.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:45:42.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:45:42.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:45:42.463 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:45:42.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:45:42.464 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:45:42.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:45:42.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:45:42.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:45:42.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:45:42.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:45:42.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:45:42.468 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:45:42.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:45:42.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:45:42.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-unexpected-clone 2024-09-18T16:45:42.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:45:42.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:45:42.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.671598 2024-09-18T16:45:42.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.671598 2024-09-18T16:45:42.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:45:42.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:45:42.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-unexpected-clone 2024-09-18T16:45:42.478 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T16:45:42.478 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:45:42.478 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.671598 2024-09-18T16:45:42.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.671598 2024-09-18T16:45:42.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T16:45:42.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T16:45:42.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T16:45:42.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T16:45:42.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-unexpected-clone 1' TERM HUP INT 2024-09-18T16:45:42.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:34: run: TEST_recover_unexpected td/osd-unexpected-clone 2024-09-18T16:45:42.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:40: TEST_recover_unexpected: local dir=td/osd-unexpected-clone 2024-09-18T16:45:42.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:42: TEST_recover_unexpected: run_mon td/osd-unexpected-clone a 2024-09-18T16:45:42.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-unexpected-clone 2024-09-18T16:45:42.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T16:45:42.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T16:45:42.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T16:45:42.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-unexpected-clone/a 2024-09-18T16:45:42.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-unexpected-clone/a --run-dir=td/osd-unexpected-clone 2024-09-18T16:45:42.862 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T16:45:42.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:45:42.863 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:45:42.863 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:45:42.863 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:45:42.864 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.671598 2024-09-18T16:45:42.864 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.671598/$cluster-$name.asok' 2024-09-18T16:45:42.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-unexpected-clone/a '--log-file=td/osd-unexpected-clone/$name.log' '--admin-socket=/tmp/ceph-asok.671598/$cluster-$name.asok' --mon-cluster-log-file=td/osd-unexpected-clone/log --run-dir=td/osd-unexpected-clone '--pid-file=td/osd-unexpected-clone/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T16:45:43.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T16:45:43.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T16:45:43.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:45:43.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:45:43.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T16:45:43.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T16:45:43.154 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:45:43.154 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:45:43.154 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:45:43.155 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:45:43.155 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:45:43.155 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.671598 2024-09-18T16:45:43.155 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.671598/ceph-mon.a.asok 2024-09-18T16:45:43.156 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:45:43.156 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.671598/ceph-mon.a.asok config get fsid 2024-09-18T16:45:43.246 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T16:45:43.247 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:45:43.247 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:45:43.247 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T16:45:43.248 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T16:45:43.248 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:45:43.248 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:45:43.248 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:45:43.248 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:45:43.248 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:45:43.248 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.671598 2024-09-18T16:45:43.249 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.671598/ceph-mon.a.asok 2024-09-18T16:45:43.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:45:43.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.671598/ceph-mon.a.asok config get mon_host 2024-09-18T16:45:43.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:43: TEST_recover_unexpected: run_mgr td/osd-unexpected-clone x 2024-09-18T16:45:43.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-unexpected-clone 2024-09-18T16:45:43.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T16:45:43.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T16:45:43.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T16:45:43.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-unexpected-clone/x 2024-09-18T16:45:43.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T16:45:44.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T16:45:44.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:45:44.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:45:44.177 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:45:44.177 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:45:44.177 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.671598 2024-09-18T16:45:44.177 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.671598/$cluster-$name.asok' 2024-09-18T16:45:44.178 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:45:44.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-unexpected-clone/x '--log-file=td/osd-unexpected-clone/$name.log' '--admin-socket=/tmp/ceph-asok.671598/$cluster-$name.asok' --run-dir=td/osd-unexpected-clone '--pid-file=td/osd-unexpected-clone/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:45:44.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:44: TEST_recover_unexpected: run_osd td/osd-unexpected-clone 0 2024-09-18T16:45:44.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-unexpected-clone 2024-09-18T16:45:44.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:45:44.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T16:45:44.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:45:44.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-unexpected-clone/0 2024-09-18T16:45:44.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3fdc75b7-dab6-4baf-bc58-f78ba81ca67a --auth-supported=none --mon-host=127.0.0.1:7144 ' 2024-09-18T16:45:44.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:45:44.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:45:44.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:45:44.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/0' 2024-09-18T16:45:44.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/0/journal' 2024-09-18T16:45:44.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:45:44.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:45:44.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2024-09-18T16:45:44.211 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:45:44.211 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:45:44.211 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:45:44.213 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:45:44.213 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:45:44.213 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.671598 2024-09-18T16:45:44.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.671598/$cluster-$name.asok' 2024-09-18T16:45:44.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.671598/$cluster-$name.asok' 2024-09-18T16:45:44.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:45:44.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:45:44.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:45:44.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2024-09-18T16:45:44.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2024-09-18T16:45:44.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:45:44.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:45:44.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:45:44.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:45:44.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:45:44.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:45:44.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-unexpected-clone/0 2024-09-18T16:45:44.215 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:45:44.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2f1bb1e0-029a-4897-a22f-1f981fe8ddc7 2024-09-18T16:45:44.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2f1bb1e0-029a-4897-a22f-1f981fe8ddc7' 2024-09-18T16:45:44.218 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 2f1bb1e0-029a-4897-a22f-1f981fe8ddc7 2024-09-18T16:45:44.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:45:44.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC4A+tmnwoFDhAADkx9Llwf8gfhHBNxhS9p7g== 2024-09-18T16:45:44.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC4A+tmnwoFDhAADkx9Llwf8gfhHBNxhS9p7g=="}' 2024-09-18T16:45:44.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2f1bb1e0-029a-4897-a22f-1f981fe8ddc7 -i td/osd-unexpected-clone/0/new.json 2024-09-18T16:45:44.545 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:45:44.561 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-unexpected-clone/0/new.json 2024-09-18T16:45:44.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=3fdc75b7-dab6-4baf-bc58-f78ba81ca67a --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/0 --osd-journal=td/osd-unexpected-clone/0/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.671598/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC4A+tmnwoFDhAADkx9Llwf8gfhHBNxhS9p7g== --osd-uuid 2f1bb1e0-029a-4897-a22f-1f981fe8ddc7 2024-09-18T16:45:44.590 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:45:44.584+0000 7fed0e889540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:45:44.594 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:45:44.588+0000 7fed0e889540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:45:44.596 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:45:44.588+0000 7fed0e889540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:45:44.597 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:45:44.592+0000 7fed0e889540 -1 bluestore(td/osd-unexpected-clone/0/block) _read_bdev_label failed to open td/osd-unexpected-clone/0/block: (2) No such file or directory 2024-09-18T16:45:44.598 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:45:44.592+0000 7fed0e889540 -1 bluestore(td/osd-unexpected-clone/0/block) _read_bdev_label failed to open td/osd-unexpected-clone/0/block: (2) No such file or directory 2024-09-18T16:45:44.598 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:45:44.592+0000 7fed0e889540 -1 bluestore(td/osd-unexpected-clone/0/block) _read_bdev_label failed to open td/osd-unexpected-clone/0/block: (2) No such file or directory 2024-09-18T16:45:44.677 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:45:44.672+0000 7fed0e889540 -1 bluestore(td/osd-unexpected-clone/0) _read_fsid unparsable uuid 2024-09-18T16:45:50.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-unexpected-clone/0/keyring 2024-09-18T16:45:50.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:45:50.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T16:45:50.336 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-09-18T16:45:50.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-unexpected-clone/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:45:50.725 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-09-18T16:45:50.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T16:45:50.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=3fdc75b7-dab6-4baf-bc58-f78ba81ca67a --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/0 --osd-journal=td/osd-unexpected-clone/0/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.671598/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:45:50.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:45:50.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:45:50.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:45:50.755 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:45:50.748+0000 7f4233f4e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:45:50.758 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:45:50.752+0000 7f4233f4e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:45:50.760 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:45:50.752+0000 7f4233f4e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:45:51.042 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:45:51.036+0000 7f4233f4e540 -1 Falling back to public interface 2024-09-18T16:45:51.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T16:45:51.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:45:51.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:45:51.073 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:45:51.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:45:51.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:45:51.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:45:51.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:45:51.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:45:51.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:45:51.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:45:52.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:45:52.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:45:52.365 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:45:52.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:45:52.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:45:52.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:45:52.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:45:53.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:45:53.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:45:53.673 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:45:53.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:45:53.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:45:53.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:45:53.691 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:45:53.684+0000 7f4233f4e540 -1 osd.0 0 log_to_monitors true 2024-09-18T16:45:53.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:45:54.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:45:55.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:45:55.296 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:45:55.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:45:55.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:45:55.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:45:55.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:45:56.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:45:56.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:45:56.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:45:56.475 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:45:56.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:45:56.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:45:56.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:45:57.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:45:57.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:45:57.788 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:45:57.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:45:57.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:45:57.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:45:58.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:45:59.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:45:59.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:45:59.095 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:45:59.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:45:59.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:45:59.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:45:59.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:00.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:00.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:00.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:46:00.408 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:46:00.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:00.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:46:00.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:01.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:01.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:01.714 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:46:01.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:46:01.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:01.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:46:02.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:03.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:03.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:03.022 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:46:03.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:46:03.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:03.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:46:03.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:04.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:04.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:04.437 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:46:04.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:46:04.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:04.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:46:04.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:05.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:05.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:05.750 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:46:05.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:46:05.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:05.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:46:06.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:07.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:07.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:07.061 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:46:07.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:46:07.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:07.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:46:07.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:08.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:08.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:08.545 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:46:08.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:46:08.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:08.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:46:09.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:10.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:10.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:10.080 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:46:10.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:46:10.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:10.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:46:10.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:11.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:11.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:11.398 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:46:11.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:46:11.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:11.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:46:11.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:13.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:13.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:13.001 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:46:13.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:46:13.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:13.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:46:13.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:14.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:14.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:14.397 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:46:14.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:46:14.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:14.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:46:14.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:15.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:15.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:15.907 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:46:15.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:46:15.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:15.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:46:16.282 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/672716,v1:127.0.0.1:6803/672716] [v2:127.0.0.1:6804/672716,v1:127.0.0.1:6805/672716] exists,up 2f1bb1e0-029a-4897-a22f-1f981fe8ddc7 2024-09-18T16:46:16.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:46:16.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:46:16.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:46:16.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:45: TEST_recover_unexpected: run_osd td/osd-unexpected-clone 1 2024-09-18T16:46:16.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-unexpected-clone 2024-09-18T16:46:16.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:46:16.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T16:46:16.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:46:16.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-unexpected-clone/1 2024-09-18T16:46:16.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3fdc75b7-dab6-4baf-bc58-f78ba81ca67a --auth-supported=none --mon-host=127.0.0.1:7144 ' 2024-09-18T16:46:16.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:46:16.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:46:16.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:46:16.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/1' 2024-09-18T16:46:16.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/1/journal' 2024-09-18T16:46:16.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:46:16.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:46:16.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2024-09-18T16:46:16.286 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:46:16.286 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:46:16.286 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:46:16.286 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:46:16.286 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:46:16.286 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.671598 2024-09-18T16:46:16.287 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.671598/$cluster-$name.asok' 2024-09-18T16:46:16.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.671598/$cluster-$name.asok' 2024-09-18T16:46:16.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:46:16.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:46:16.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:46:16.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2024-09-18T16:46:16.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2024-09-18T16:46:16.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:46:16.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:46:16.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:46:16.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:46:16.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:46:16.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:46:16.288 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-unexpected-clone/1 2024-09-18T16:46:16.289 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:46:16.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f1536932-bcaf-48b5-8d58-3dab1a193430 2024-09-18T16:46:16.289 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 f1536932-bcaf-48b5-8d58-3dab1a193430 2024-09-18T16:46:16.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f1536932-bcaf-48b5-8d58-3dab1a193430' 2024-09-18T16:46:16.290 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:46:16.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDYA+tmsQhEEhAAp/0V954PJLojmvNZCcDhqw== 2024-09-18T16:46:16.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDYA+tmsQhEEhAAp/0V954PJLojmvNZCcDhqw=="}' 2024-09-18T16:46:16.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f1536932-bcaf-48b5-8d58-3dab1a193430 -i td/osd-unexpected-clone/1/new.json 2024-09-18T16:46:16.761 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:46:16.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-unexpected-clone/1/new.json 2024-09-18T16:46:16.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=3fdc75b7-dab6-4baf-bc58-f78ba81ca67a --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/1 --osd-journal=td/osd-unexpected-clone/1/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.671598/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDYA+tmsQhEEhAAp/0V954PJLojmvNZCcDhqw== --osd-uuid f1536932-bcaf-48b5-8d58-3dab1a193430 2024-09-18T16:46:16.812 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:46:16.804+0000 7f1e16ffd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:46:16.817 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:46:16.812+0000 7f1e16ffd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:46:16.819 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:46:16.812+0000 7f1e16ffd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:46:16.820 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:46:16.812+0000 7f1e16ffd540 -1 bluestore(td/osd-unexpected-clone/1/block) _read_bdev_label failed to open td/osd-unexpected-clone/1/block: (2) No such file or directory 2024-09-18T16:46:16.820 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:46:16.812+0000 7f1e16ffd540 -1 bluestore(td/osd-unexpected-clone/1/block) _read_bdev_label failed to open td/osd-unexpected-clone/1/block: (2) No such file or directory 2024-09-18T16:46:16.820 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:46:16.812+0000 7f1e16ffd540 -1 bluestore(td/osd-unexpected-clone/1/block) _read_bdev_label failed to open td/osd-unexpected-clone/1/block: (2) No such file or directory 2024-09-18T16:46:16.882 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:46:16.876+0000 7f1e16ffd540 -1 bluestore(td/osd-unexpected-clone/1) _read_fsid unparsable uuid 2024-09-18T16:46:22.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-unexpected-clone/1/keyring 2024-09-18T16:46:22.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:46:22.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T16:46:22.805 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-09-18T16:46:22.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-unexpected-clone/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:46:23.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T16:46:23.190 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-09-18T16:46:23.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=3fdc75b7-dab6-4baf-bc58-f78ba81ca67a --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/1 --osd-journal=td/osd-unexpected-clone/1/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.671598/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:46:23.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:46:23.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:46:23.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:46:23.213 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:46:23.208+0000 7f37d968e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:46:23.217 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:46:23.212+0000 7f37d968e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:46:23.226 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:46:23.220+0000 7f37d968e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:46:23.511 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:46:23.504+0000 7f37d968e540 -1 Falling back to public interface 2024-09-18T16:46:23.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T16:46:23.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:46:23.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T16:46:23.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:46:23.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:46:23.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:23.546 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:46:23.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:46:23.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:23.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:46:23.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:24.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:24.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:24.846 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:46:24.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:46:24.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:24.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:46:25.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:26.030 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:46:26.024+0000 7f37d968e540 -1 osd.1 0 log_to_monitors true 2024-09-18T16:46:26.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:26.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:26.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:46:26.156 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:46:26.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:26.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:46:26.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:27.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:27.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:27.567 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:46:27.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:46:27.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:27.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:46:28.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:29.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:29.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:29.002 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:46:29.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:46:29.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:29.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:46:29.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:30.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:30.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:30.322 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:46:30.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:46:30.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:30.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:46:30.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:31.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:31.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:31.651 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:46:31.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:46:31.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:31.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:46:31.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:32.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:32.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:32.972 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:46:32.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:46:32.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:32.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:46:33.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:34.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:34.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:34.269 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:46:34.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:46:34.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:34.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:46:34.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:35.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:35.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:35.601 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:46:35.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:46:35.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:35.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:46:35.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:36.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:36.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:36.946 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:46:36.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:46:36.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:36.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:46:37.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:38.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:38.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:38.258 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:46:38.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:46:38.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:38.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:46:38.939 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:39.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:39.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:39.941 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:46:39.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:46:39.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:39.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:46:40.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:41.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:41.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:41.319 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:46:41.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:46:41.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:41.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:46:41.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:42.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:42.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:42.672 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:46:42.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:46:42.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:42.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:46:43.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:44.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:44.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:44.352 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:46:44.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:46:44.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:44.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:46:44.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:45.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:45.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:45.836 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:46:45.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:46:45.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:45.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:46:46.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:47.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:47.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:47.380 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:46:47.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:46:47.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:47.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:46:48.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:49.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:49.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:49.042 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:46:49.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:46:49.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:49.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:46:49.428 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/674495,v1:127.0.0.1:6811/674495] [v2:127.0.0.1:6812/674495,v1:127.0.0.1:6813/674495] exists,up f1536932-bcaf-48b5-8d58-3dab1a193430 2024-09-18T16:46:49.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:46:49.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:46:49.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:46:49.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:46: TEST_recover_unexpected: run_osd td/osd-unexpected-clone 2 2024-09-18T16:46:49.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-unexpected-clone 2024-09-18T16:46:49.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:46:49.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T16:46:49.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:46:49.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-unexpected-clone/2 2024-09-18T16:46:49.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3fdc75b7-dab6-4baf-bc58-f78ba81ca67a --auth-supported=none --mon-host=127.0.0.1:7144 ' 2024-09-18T16:46:49.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:46:49.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:46:49.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:46:49.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/2' 2024-09-18T16:46:49.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/2/journal' 2024-09-18T16:46:49.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:46:49.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:46:49.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2024-09-18T16:46:49.433 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:46:49.433 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:46:49.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:46:49.434 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:46:49.434 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:46:49.434 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.671598 2024-09-18T16:46:49.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.671598/$cluster-$name.asok' 2024-09-18T16:46:49.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.671598/$cluster-$name.asok' 2024-09-18T16:46:49.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:46:49.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:46:49.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:46:49.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2024-09-18T16:46:49.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2024-09-18T16:46:49.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:46:49.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:46:49.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:46:49.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:46:49.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:46:49.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:46:49.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-unexpected-clone/2 2024-09-18T16:46:49.436 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:46:49.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=188f8efd-f9a6-40e9-8807-17d49af0200d 2024-09-18T16:46:49.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 188f8efd-f9a6-40e9-8807-17d49af0200d' 2024-09-18T16:46:49.438 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 188f8efd-f9a6-40e9-8807-17d49af0200d 2024-09-18T16:46:49.438 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:46:49.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD5A+tmclIWGxAAld5cBk7MDzXYugxSZPAdNw== 2024-09-18T16:46:49.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD5A+tmclIWGxAAld5cBk7MDzXYugxSZPAdNw=="}' 2024-09-18T16:46:49.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 188f8efd-f9a6-40e9-8807-17d49af0200d -i td/osd-unexpected-clone/2/new.json 2024-09-18T16:46:49.836 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:46:49.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-unexpected-clone/2/new.json 2024-09-18T16:46:49.857 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=3fdc75b7-dab6-4baf-bc58-f78ba81ca67a --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/2 --osd-journal=td/osd-unexpected-clone/2/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.671598/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD5A+tmclIWGxAAld5cBk7MDzXYugxSZPAdNw== --osd-uuid 188f8efd-f9a6-40e9-8807-17d49af0200d 2024-09-18T16:46:49.889 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:46:49.888+0000 7fb945712540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:46:49.911 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:46:49.908+0000 7fb945712540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:46:49.913 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:46:49.912+0000 7fb945712540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:46:49.914 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:46:49.912+0000 7fb945712540 -1 bluestore(td/osd-unexpected-clone/2/block) _read_bdev_label failed to open td/osd-unexpected-clone/2/block: (2) No such file or directory 2024-09-18T16:46:49.914 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:46:49.912+0000 7fb945712540 -1 bluestore(td/osd-unexpected-clone/2/block) _read_bdev_label failed to open td/osd-unexpected-clone/2/block: (2) No such file or directory 2024-09-18T16:46:49.914 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:46:49.912+0000 7fb945712540 -1 bluestore(td/osd-unexpected-clone/2/block) _read_bdev_label failed to open td/osd-unexpected-clone/2/block: (2) No such file or directory 2024-09-18T16:46:49.992 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:46:49.988+0000 7fb945712540 -1 bluestore(td/osd-unexpected-clone/2) _read_fsid unparsable uuid 2024-09-18T16:46:56.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-unexpected-clone/2/keyring 2024-09-18T16:46:56.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:46:56.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T16:46:56.217 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-09-18T16:46:56.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-unexpected-clone/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:46:56.653 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-09-18T16:46:56.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T16:46:56.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=3fdc75b7-dab6-4baf-bc58-f78ba81ca67a --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/2 --osd-journal=td/osd-unexpected-clone/2/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.671598/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:46:56.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:46:56.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:46:56.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:46:56.684 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:46:56.680+0000 7f0013620540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:46:56.686 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:46:56.684+0000 7f0013620540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:46:56.688 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:46:56.684+0000 7f0013620540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:46:56.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T16:46:56.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:46:56.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T16:46:56.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:46:56.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:46:56.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:56.955 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-09-18T16:46:56.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:46:56.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:56.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:46:57.244 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:46:57.240+0000 7f0013620540 -1 Falling back to public interface 2024-09-18T16:46:57.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:58.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:58.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:58.310 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-09-18T16:46:58.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:46:58.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:58.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:46:58.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:59.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:46:59.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:46:59.621 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-09-18T16:46:59.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:46:59.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:46:59.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:46:59.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:46:59.949 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:46:59.947+0000 7f0013620540 -1 osd.2 0 log_to_monitors true 2024-09-18T16:47:00.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:47:00.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:00.933 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-09-18T16:47:00.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:47:00.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:00.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:47:01.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:47:02.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:47:02.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:02.242 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-09-18T16:47:02.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:47:02.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:02.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:47:02.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:47:03.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:47:03.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:03.546 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-09-18T16:47:03.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:47:03.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:03.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:47:04.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:47:05.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:47:05.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:05.026 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-09-18T16:47:05.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:47:05.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:05.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:47:05.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:47:06.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:47:06.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:06.335 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-09-18T16:47:06.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:47:06.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:06.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:47:06.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:47:07.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:47:07.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:07.710 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-09-18T16:47:07.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:47:07.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:07.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:47:08.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:47:09.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:47:09.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:09.074 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-09-18T16:47:09.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:47:09.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:09.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:47:09.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:47:10.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:47:10.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:10.393 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-09-18T16:47:10.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:47:10.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:10.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:47:10.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:47:11.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:47:11.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:11.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:47:11.913 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-09-18T16:47:11.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:11.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:47:12.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:47:13.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:47:13.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:13.224 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-09-18T16:47:13.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:47:13.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:13.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:47:13.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:47:14.622 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:47:14.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:14.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:47:14.623 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-09-18T16:47:14.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:14.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:47:15.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:47:16.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:47:16.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:16.334 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-09-18T16:47:16.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:47:16.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:16.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:47:16.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:47:17.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:47:17.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:17.736 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-09-18T16:47:17.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:47:17.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:17.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:47:18.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:47:19.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:47:19.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:19.360 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-09-18T16:47:19.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:47:19.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:19.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:47:20.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:47:21.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:47:21.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:21.078 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-09-18T16:47:21.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:47:21.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:21.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:47:21.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:47:22.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:47:22.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:22.667 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-09-18T16:47:22.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T16:47:22.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:22.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:47:23.036 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/676274,v1:127.0.0.1:6819/676274] [v2:127.0.0.1:6820/676274,v1:127.0.0.1:6821/676274] exists,up 188f8efd-f9a6-40e9-8807-17d49af0200d 2024-09-18T16:47:23.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:47:23.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:47:23.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:47:23.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:48: TEST_recover_unexpected: ceph osd pool create foo 1 2024-09-18T16:47:23.479 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'foo' created 2024-09-18T16:47:23.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:49: TEST_recover_unexpected: rados -p foo put foo /etc/passwd 2024-09-18T16:47:24.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:50: TEST_recover_unexpected: rados -p foo mksnap snap 2024-09-18T16:47:25.128 INFO:tasks.workunit.client.0.smithi046.stdout:created pool foo snap snap 2024-09-18T16:47:25.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:51: TEST_recover_unexpected: rados -p foo put foo /etc/group 2024-09-18T16:47:25.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:53: TEST_recover_unexpected: wait_for_clean 2024-09-18T16:47:25.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:47:25.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:47:25.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:47:25.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:47:25.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:47:25.369 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:47:25.369 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:47:25.370 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:47:25.370 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:47:25.370 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:47:25.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:47:25.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:47:25.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:47:25.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:47:25.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:47:25.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:47:25.895 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:47:25.895 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:47:25.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:47:25.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:47:25.895 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:47:26.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836498 2024-09-18T16:47:26.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836498 2024-09-18T16:47:26.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498' 2024-09-18T16:47:26.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:47:26.186 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:47:26.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672972 2024-09-18T16:47:26.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672972 2024-09-18T16:47:26.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498 1-42949672972' 2024-09-18T16:47:26.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:47:26.349 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:47:26.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509448 2024-09-18T16:47:26.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509448 2024-09-18T16:47:26.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498 1-42949672972 2-64424509448' 2024-09-18T16:47:26.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:47:26.491 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836498 2024-09-18T16:47:26.491 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:47:26.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:47:26.494 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:47:26.494 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836498 2024-09-18T16:47:26.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836498 2024-09-18T16:47:26.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-09-18T16:47:26.496 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836498 2024-09-18T16:47:26.496 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:47:26.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836497 -lt 21474836498 2024-09-18T16:47:26.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:47:27.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:47:27.801 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:47:28.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836499 -lt 21474836498 2024-09-18T16:47:28.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:47:28.142 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672972 2024-09-18T16:47:28.142 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:47:28.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:47:28.146 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672972 2024-09-18T16:47:28.146 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:47:28.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672972 2024-09-18T16:47:28.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-09-18T16:47:28.147 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672972 2024-09-18T16:47:28.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:47:28.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672972 -lt 42949672972 2024-09-18T16:47:28.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:47:28.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509448 2024-09-18T16:47:28.453 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:47:28.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:47:28.457 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509448 2024-09-18T16:47:28.457 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:47:28.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509448 2024-09-18T16:47:28.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-18T16:47:28.458 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509448 2024-09-18T16:47:28.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:47:28.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509448 -lt 64424509448 2024-09-18T16:47:28.758 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:47:28.759 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:47:28.759 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:47:29.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:47:29.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:47:29.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:47:29.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:47:29.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:47:29.105 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:47:29.106 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:47:29.106 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:47:29.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:47:29.449 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:47:29.450 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:47:29.450 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:47:29.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:47:29.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:47:29.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:47:29.819 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:55: TEST_recover_unexpected: get_primary foo foo 2024-09-18T16:47:29.819 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=foo 2024-09-18T16:47:29.820 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=foo 2024-09-18T16:47:29.820 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map foo foo 2024-09-18T16:47:29.820 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T16:47:30.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:55: TEST_recover_unexpected: local osd=1 2024-09-18T16:47:30.152 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:57: TEST_recover_unexpected: objectstore_tool td/osd-unexpected-clone 1 --op list foo 2024-09-18T16:47:30.152 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-unexpected-clone 2024-09-18T16:47:30.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T16:47:30.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:57: TEST_recover_unexpected: grep snapid.:1 2024-09-18T16:47:30.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T16:47:30.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T16:47:30.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-unexpected-clone 1 --op list foo 2024-09-18T16:47:30.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-unexpected-clone 2024-09-18T16:47:30.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T16:47:30.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T16:47:30.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T16:47:30.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-unexpected-clone TERM osd.1 2024-09-18T16:47:30.154 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:47:30.154 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:47:30.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:47:30.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:47:30.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:47:30.465 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:47:30.465 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-unexpected-clone 1 --op list foo 2024-09-18T16:47:30.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-unexpected-clone 2024-09-18T16:47:30.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T16:47:30.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T16:47:30.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T16:47:30.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-unexpected-clone/1 2024-09-18T16:47:30.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-unexpected-clone/1 --op list foo 2024-09-18T16:47:32.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-unexpected-clone 1 2024-09-18T16:47:32.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-unexpected-clone 2024-09-18T16:47:32.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T16:47:32.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T16:47:32.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T16:47:32.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-unexpected-clone/1 2024-09-18T16:47:32.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3fdc75b7-dab6-4baf-bc58-f78ba81ca67a --auth-supported=none --mon-host=127.0.0.1:7144 ' 2024-09-18T16:47:32.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:47:32.624 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:47:32.625 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:47:32.625 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/1' 2024-09-18T16:47:32.625 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/1/journal' 2024-09-18T16:47:32.625 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T16:47:32.625 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T16:47:32.625 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2024-09-18T16:47:32.625 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T16:47:32.625 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:47:32.625 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:47:32.627 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:47:32.627 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:47:32.627 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.671598 2024-09-18T16:47:32.627 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.671598/$cluster-$name.asok' 2024-09-18T16:47:32.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.671598/$cluster-$name.asok' 2024-09-18T16:47:32.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:47:32.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2024-09-18T16:47:32.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2024-09-18T16:47:32.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:47:32.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:47:32.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:47:32.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:47:32.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T16:47:32.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T16:47:32.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-unexpected-clone/1 2024-09-18T16:47:32.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T16:47:32.628 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T16:47:32.629 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=3fdc75b7-dab6-4baf-bc58-f78ba81ca67a --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/1 --osd-journal=td/osd-unexpected-clone/1/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.671598/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:47:32.629 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-unexpected-clone/1/whoami 2024-09-18T16:47:32.631 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T16:47:32.631 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T16:47:32.632 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T16:47:32.632 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T16:47:32.659 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:47:32.655+0000 7fbbf6c99540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:47:32.662 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:47:32.659+0000 7fbbf6c99540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:47:32.670 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:47:32.667+0000 7fbbf6c99540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:47:32.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T16:47:32.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:47:32.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T16:47:32.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:47:32.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:47:32.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:32.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:47:32.933 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T16:47:32.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:32.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:47:33.208 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:47:33.203+0000 7fbbf6c99540 -1 Falling back to public interface 2024-09-18T16:47:33.274 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:47:34.276 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:47:34.276 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:34.276 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:47:34.276 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:47:34.276 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:34.277 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:47:34.583 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:47:35.586 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:47:35.740 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:35.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:47:35.741 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T16:47:35.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:35.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:47:35.906 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:47:35.917 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:47:35.915+0000 7fbbf6c99540 -1 osd.1 20 log_to_monitors true 2024-09-18T16:47:36.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:47:36.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:36.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:47:36.908 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T16:47:36.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:36.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:47:37.219 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 24 up_thru 24 down_at 21 last_clean_interval [10,20) [v2:127.0.0.1:6810/678117,v1:127.0.0.1:6811/678117] [v2:127.0.0.1:6812/678117,v1:127.0.0.1:6813/678117] exists,up f1536932-bcaf-48b5-8d58-3dab1a193430 2024-09-18T16:47:37.219 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:47:37.219 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:47:37.219 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:47:37.219 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T16:47:37.219 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:47:37.219 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:47:37.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:47:37.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:47:37.220 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:47:37.220 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:47:37.220 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:47:37.221 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:47:37.221 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:47:37.221 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:47:37.428 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:47:37.428 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:47:37.428 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:47:37.428 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:47:37.429 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:47:37.740 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:47:37.740 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:47:37.740 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:47:37.740 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:47:37.740 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:47:37.740 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:47:37.880 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836502 2024-09-18T16:47:38.080 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836502 2024-09-18T16:47:38.080 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836502' 2024-09-18T16:47:38.080 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:47:38.081 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:47:38.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=103079215106 2024-09-18T16:47:38.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 103079215106 2024-09-18T16:47:38.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836502 1-103079215106' 2024-09-18T16:47:38.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:47:38.082 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:47:38.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509451 2024-09-18T16:47:38.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509451 2024-09-18T16:47:38.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836502 1-103079215106 2-64424509451' 2024-09-18T16:47:38.149 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:47:38.149 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836502 2024-09-18T16:47:38.149 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:47:38.151 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:47:38.152 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836502 2024-09-18T16:47:38.152 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:47:38.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836502 2024-09-18T16:47:38.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-18T16:47:38.154 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836502 2024-09-18T16:47:38.154 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:47:38.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836501 -lt 21474836502 2024-09-18T16:47:38.462 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:47:39.463 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:47:39.464 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:47:39.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836502 -lt 21474836502 2024-09-18T16:47:39.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:47:39.792 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-103079215106 2024-09-18T16:47:39.793 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:47:39.794 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:47:39.795 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-103079215106 2024-09-18T16:47:39.796 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:47:39.798 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=103079215106 2024-09-18T16:47:39.798 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 103079215106' 2024-09-18T16:47:39.798 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 103079215106 2024-09-18T16:47:39.798 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:47:40.093 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 103079215106 -lt 103079215106 2024-09-18T16:47:40.093 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:47:40.093 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509451 2024-09-18T16:47:40.093 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:47:40.095 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:47:40.096 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509451 2024-09-18T16:47:40.096 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:47:40.098 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509451 2024-09-18T16:47:40.098 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-18T16:47:40.098 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509451 2024-09-18T16:47:40.099 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:47:40.438 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509452 -lt 64424509451 2024-09-18T16:47:40.439 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:47:40.439 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:47:40.440 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:47:40.791 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:47:40.791 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:47:40.792 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:47:40.793 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:47:40.793 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:47:40.793 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:47:40.793 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:47:40.793 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:47:41.133 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:47:41.133 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:47:41.134 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:47:41.134 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:47:41.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:47:41.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:47:41.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:47:41.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:57: TEST_recover_unexpected: JSON='["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:47:41.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:58: TEST_recover_unexpected: echo 'JSON is ["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-18T16:47:41.490 INFO:tasks.workunit.client.0.smithi046.stdout:JSON is ["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}] 2024-09-18T16:47:41.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:59: TEST_recover_unexpected: rm -f td/osd-unexpected-clone/_ td/osd-unexpected-clone/data 2024-09-18T16:47:41.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:60: TEST_recover_unexpected: objectstore_tool td/osd-unexpected-clone 1 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-attr _ 2024-09-18T16:47:41.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-unexpected-clone 2024-09-18T16:47:41.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T16:47:41.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T16:47:41.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T16:47:41.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-unexpected-clone 1 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-attr _ 2024-09-18T16:47:41.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-unexpected-clone 2024-09-18T16:47:41.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T16:47:41.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T16:47:41.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T16:47:41.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-unexpected-clone TERM osd.1 2024-09-18T16:47:41.494 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:47:41.494 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:47:41.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:47:41.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:47:41.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:47:41.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:47:41.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-unexpected-clone 1 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-attr _ 2024-09-18T16:47:41.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-unexpected-clone 2024-09-18T16:47:41.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T16:47:41.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T16:47:41.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T16:47:41.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-unexpected-clone/1 2024-09-18T16:47:41.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-unexpected-clone/1 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-attr _ 2024-09-18T16:47:43.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-unexpected-clone 1 2024-09-18T16:47:43.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-unexpected-clone 2024-09-18T16:47:43.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T16:47:43.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T16:47:43.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T16:47:43.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-unexpected-clone/1 2024-09-18T16:47:43.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3fdc75b7-dab6-4baf-bc58-f78ba81ca67a --auth-supported=none --mon-host=127.0.0.1:7144 ' 2024-09-18T16:47:43.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:47:43.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:47:43.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:47:43.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/1' 2024-09-18T16:47:43.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/1/journal' 2024-09-18T16:47:43.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T16:47:43.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T16:47:43.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2024-09-18T16:47:43.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T16:47:43.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:47:43.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:47:43.813 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:47:43.813 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:47:43.813 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.671598 2024-09-18T16:47:43.814 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.671598/$cluster-$name.asok' 2024-09-18T16:47:43.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.671598/$cluster-$name.asok' 2024-09-18T16:47:43.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:47:43.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2024-09-18T16:47:43.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2024-09-18T16:47:43.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:47:43.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:47:43.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:47:43.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:47:43.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T16:47:43.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T16:47:43.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-unexpected-clone/1 2024-09-18T16:47:43.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T16:47:43.816 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.1 2024-09-18T16:47:43.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=3fdc75b7-dab6-4baf-bc58-f78ba81ca67a --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/1 --osd-journal=td/osd-unexpected-clone/1/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.671598/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:47:43.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-unexpected-clone/1/whoami 2024-09-18T16:47:43.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T16:47:43.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T16:47:43.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T16:47:43.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T16:47:43.841 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:47:43.839+0000 7f37bc446540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:47:43.844 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:47:43.839+0000 7f37bc446540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:47:43.846 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:47:43.843+0000 7f37bc446540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:47:44.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T16:47:44.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:47:44.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T16:47:44.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:47:44.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:47:44.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:44.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:47:44.126 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T16:47:44.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:44.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:47:44.379 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:47:44.375+0000 7f37bc446540 -1 Falling back to public interface 2024-09-18T16:47:44.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:47:45.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:47:45.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:45.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:47:45.476 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:47:45.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:45.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:47:45.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:47:46.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:47:46.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:46.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:47:46.822 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T16:47:46.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:46.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:47:46.963 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:47:46.959+0000 7f37bc446540 -1 osd.1 25 log_to_monitors true 2024-09-18T16:47:47.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:47:48.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:47:48.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:48.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:47:48.191 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T16:47:48.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:48.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:47:48.550 INFO:tasks.workunit.client.0.smithi046.stderr:osd.1 up in weight 1 up_from 29 up_thru 29 down_at 26 last_clean_interval [24,25) [v2:127.0.0.1:6810/679472,v1:127.0.0.1:6811/679472] [v2:127.0.0.1:6812/679472,v1:127.0.0.1:6813/679472] exists,up f1536932-bcaf-48b5-8d58-3dab1a193430 2024-09-18T16:47:48.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:47:48.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:47:48.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:47:48.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T16:47:48.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:47:48.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:47:48.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:47:48.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:47:48.551 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:47:48.551 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:47:48.551 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:47:48.551 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:47:48.552 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:47:48.552 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:47:48.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:47:48.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:47:48.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:47:48.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:47:48.754 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:47:49.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:47:49.071 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:47:49.071 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:47:49.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:47:49.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:47:49.071 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:47:49.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836505 2024-09-18T16:47:49.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836505 2024-09-18T16:47:49.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505' 2024-09-18T16:47:49.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:47:49.240 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:47:49.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051586 2024-09-18T16:47:49.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051586 2024-09-18T16:47:49.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-124554051586' 2024-09-18T16:47:49.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:47:49.378 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:47:49.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509455 2024-09-18T16:47:49.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509455 2024-09-18T16:47:49.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-124554051586 2-64424509455' 2024-09-18T16:47:49.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:47:49.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836505 2024-09-18T16:47:49.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:47:49.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:47:49.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836505 2024-09-18T16:47:49.514 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:47:49.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836505 2024-09-18T16:47:49.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-18T16:47:49.516 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836505 2024-09-18T16:47:49.517 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:47:49.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836505 -lt 21474836505 2024-09-18T16:47:49.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:47:49.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-124554051586 2024-09-18T16:47:49.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:47:49.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:47:49.824 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-124554051586 2024-09-18T16:47:49.824 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:47:49.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051586 2024-09-18T16:47:49.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2024-09-18T16:47:49.826 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 124554051586 2024-09-18T16:47:49.827 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:47:50.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051586 -lt 124554051586 2024-09-18T16:47:50.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:47:50.128 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509455 2024-09-18T16:47:50.128 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:47:50.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:47:50.130 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509455 2024-09-18T16:47:50.130 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:47:50.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509455 2024-09-18T16:47:50.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-18T16:47:50.132 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 64424509455 2024-09-18T16:47:50.132 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:47:50.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-18T16:47:50.468 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:47:50.469 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:47:50.469 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:47:50.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:47:50.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:47:50.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:47:50.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:47:50.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:47:50.823 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:47:50.824 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:47:50.824 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:47:51.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:47:51.170 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:47:51.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:47:51.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:47:51.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:47:51.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:47:51.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:47:51.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:61: TEST_recover_unexpected: objectstore_tool td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-bytes td/osd-unexpected-clone/data 2024-09-18T16:47:51.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-unexpected-clone 2024-09-18T16:47:51.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T16:47:51.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-18T16:47:51.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T16:47:51.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-bytes td/osd-unexpected-clone/data 2024-09-18T16:47:51.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-unexpected-clone 2024-09-18T16:47:51.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T16:47:51.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-18T16:47:51.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T16:47:51.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-unexpected-clone TERM osd.2 2024-09-18T16:47:51.542 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:47:51.542 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:47:51.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:47:51.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:47:51.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:47:51.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:47:51.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-bytes td/osd-unexpected-clone/data 2024-09-18T16:47:51.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-unexpected-clone 2024-09-18T16:47:51.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T16:47:51.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-18T16:47:51.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T16:47:51.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-unexpected-clone/2 2024-09-18T16:47:51.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-unexpected-clone/2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-bytes td/osd-unexpected-clone/data 2024-09-18T16:47:54.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-unexpected-clone 2 2024-09-18T16:47:54.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-unexpected-clone 2024-09-18T16:47:54.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T16:47:54.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T16:47:54.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T16:47:54.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-unexpected-clone/2 2024-09-18T16:47:54.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3fdc75b7-dab6-4baf-bc58-f78ba81ca67a --auth-supported=none --mon-host=127.0.0.1:7144 ' 2024-09-18T16:47:54.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:47:54.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:47:54.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:47:54.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/2' 2024-09-18T16:47:54.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/2/journal' 2024-09-18T16:47:54.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T16:47:54.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T16:47:54.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2024-09-18T16:47:54.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T16:47:54.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:47:54.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:47:54.055 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:47:54.055 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:47:54.055 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.671598 2024-09-18T16:47:54.055 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.671598/$cluster-$name.asok' 2024-09-18T16:47:54.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.671598/$cluster-$name.asok' 2024-09-18T16:47:54.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:47:54.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2024-09-18T16:47:54.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2024-09-18T16:47:54.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:47:54.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:47:54.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:47:54.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:47:54.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T16:47:54.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T16:47:54.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-unexpected-clone/2 2024-09-18T16:47:54.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T16:47:54.057 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.2 2024-09-18T16:47:54.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=3fdc75b7-dab6-4baf-bc58-f78ba81ca67a --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/2 --osd-journal=td/osd-unexpected-clone/2/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.671598/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:47:54.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-unexpected-clone/2/whoami 2024-09-18T16:47:54.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T16:47:54.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T16:47:54.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T16:47:54.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T16:47:54.087 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:47:54.083+0000 7fa717fa3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:47:54.090 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:47:54.087+0000 7fa717fa3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:47:54.097 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:47:54.087+0000 7fa717fa3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:47:54.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T16:47:54.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:47:54.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T16:47:54.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:47:54.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:47:54.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:54.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:47:54.406 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T16:47:54.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:54.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:47:54.623 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:47:54.619+0000 7fa717fa3540 -1 Falling back to public interface 2024-09-18T16:47:54.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:47:55.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:47:55.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:55.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:47:55.760 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:47:55.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:55.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:47:56.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:47:57.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:47:57.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:57.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:47:57.070 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T16:47:57.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:57.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:47:57.199 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:47:57.195+0000 7fa717fa3540 -1 osd.2 30 log_to_monitors true 2024-09-18T16:47:57.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:47:58.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:47:58.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:47:58.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:47:58.479 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T16:47:58.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:47:58.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:47:58.849 INFO:tasks.workunit.client.0.smithi046.stderr:osd.2 up in weight 1 up_from 34 up_thru 0 down_at 31 last_clean_interval [15,30) [v2:127.0.0.1:6818/680802,v1:127.0.0.1:6819/680802] [v2:127.0.0.1:6820/680802,v1:127.0.0.1:6821/680802] exists,up 188f8efd-f9a6-40e9-8807-17d49af0200d 2024-09-18T16:47:58.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:47:58.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:47:58.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:47:58.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T16:47:58.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:47:58.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:47:58.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:47:58.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:47:58.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:47:58.851 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:47:58.851 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:47:58.851 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:47:58.851 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:47:58.851 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:47:59.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:47:59.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:47:59.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:47:59.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:47:59.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:47:59.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:47:59.364 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:47:59.364 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:47:59.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:47:59.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:47:59.365 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:47:59.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836508 2024-09-18T16:47:59.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836508 2024-09-18T16:47:59.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836508' 2024-09-18T16:47:59.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:47:59.540 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:47:59.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051589 2024-09-18T16:47:59.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051589 2024-09-18T16:47:59.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836508 1-124554051589' 2024-09-18T16:47:59.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:47:59.676 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:47:59.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=146028888066 2024-09-18T16:47:59.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 146028888066 2024-09-18T16:47:59.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836508 1-124554051589 2-146028888066' 2024-09-18T16:47:59.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:47:59.927 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836508 2024-09-18T16:47:59.928 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:47:59.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:47:59.928 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836508 2024-09-18T16:47:59.928 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:47:59.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836508 2024-09-18T16:47:59.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-09-18T16:47:59.928 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836508 2024-09-18T16:47:59.928 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:48:00.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836508 -lt 21474836508 2024-09-18T16:48:00.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:48:00.109 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-124554051589 2024-09-18T16:48:00.110 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:48:00.112 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:48:00.113 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-124554051589 2024-09-18T16:48:00.113 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:48:00.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051589 2024-09-18T16:48:00.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 124554051589' 2024-09-18T16:48:00.116 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 124554051589 2024-09-18T16:48:00.116 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:48:00.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051588 -lt 124554051589 2024-09-18T16:48:00.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:48:01.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:48:01.454 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:48:01.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051589 -lt 124554051589 2024-09-18T16:48:01.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:48:01.787 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-146028888066 2024-09-18T16:48:01.787 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:48:01.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:48:01.789 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-146028888066 2024-09-18T16:48:01.789 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:48:01.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=146028888066 2024-09-18T16:48:01.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 146028888066' 2024-09-18T16:48:01.791 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 146028888066 2024-09-18T16:48:01.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:48:02.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 146028888066 -lt 146028888066 2024-09-18T16:48:02.135 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:48:02.136 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:48:02.137 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:48:02.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:48:02.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:48:02.542 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:48:02.542 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:48:02.542 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:48:02.542 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:48:02.542 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:48:02.542 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:48:02.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:48:02.881 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:48:02.882 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:48:02.882 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:48:03.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:48:03.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:48:03.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:48:03.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:63: TEST_recover_unexpected: rados -p foo rmsnap snap 2024-09-18T16:48:03.440 INFO:tasks.workunit.client.0.smithi046.stdout:removed pool foo snap snap 2024-09-18T16:48:03.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:65: TEST_recover_unexpected: sleep 5 2024-09-18T16:48:08.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:67: TEST_recover_unexpected: objectstore_tool td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes td/osd-unexpected-clone/data 2024-09-18T16:48:08.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-unexpected-clone 2024-09-18T16:48:08.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T16:48:08.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-18T16:48:08.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T16:48:08.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes td/osd-unexpected-clone/data 2024-09-18T16:48:08.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-unexpected-clone 2024-09-18T16:48:08.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T16:48:08.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-18T16:48:08.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T16:48:08.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-unexpected-clone TERM osd.2 2024-09-18T16:48:08.449 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:48:08.449 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:48:08.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:48:08.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:48:08.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:48:08.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:48:08.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes td/osd-unexpected-clone/data 2024-09-18T16:48:08.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-unexpected-clone 2024-09-18T16:48:08.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T16:48:08.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-18T16:48:08.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T16:48:08.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-unexpected-clone/2 2024-09-18T16:48:08.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-unexpected-clone/2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes td/osd-unexpected-clone/data 2024-09-18T16:48:11.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-unexpected-clone 2 2024-09-18T16:48:11.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-unexpected-clone 2024-09-18T16:48:11.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T16:48:11.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T16:48:11.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T16:48:11.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-unexpected-clone/2 2024-09-18T16:48:11.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3fdc75b7-dab6-4baf-bc58-f78ba81ca67a --auth-supported=none --mon-host=127.0.0.1:7144 ' 2024-09-18T16:48:11.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:48:11.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:48:11.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:48:11.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/2' 2024-09-18T16:48:11.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/2/journal' 2024-09-18T16:48:11.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T16:48:11.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T16:48:11.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2024-09-18T16:48:11.080 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T16:48:11.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:48:11.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:48:11.082 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:48:11.082 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:48:11.082 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.671598 2024-09-18T16:48:11.083 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.671598/$cluster-$name.asok' 2024-09-18T16:48:11.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.671598/$cluster-$name.asok' 2024-09-18T16:48:11.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:48:11.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2024-09-18T16:48:11.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2024-09-18T16:48:11.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:48:11.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:48:11.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:48:11.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:48:11.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T16:48:11.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T16:48:11.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-unexpected-clone/2 2024-09-18T16:48:11.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T16:48:11.084 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.2 2024-09-18T16:48:11.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=3fdc75b7-dab6-4baf-bc58-f78ba81ca67a --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/2 --osd-journal=td/osd-unexpected-clone/2/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.671598/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:48:11.084 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-unexpected-clone/2/whoami 2024-09-18T16:48:11.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T16:48:11.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T16:48:11.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T16:48:11.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T16:48:11.114 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:48:11.111+0000 7f5b63531540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:48:11.118 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:48:11.115+0000 7f5b63531540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:48:11.120 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:48:11.115+0000 7f5b63531540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:48:11.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T16:48:11.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:48:11.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T16:48:11.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:48:11.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:48:11.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:48:11.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:48:11.404 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T16:48:11.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:48:11.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:48:11.663 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:48:11.659+0000 7f5b63531540 -1 Falling back to public interface 2024-09-18T16:48:11.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:48:12.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:48:12.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:48:12.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:48:12.740 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:48:12.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:48:12.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:48:13.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:48:14.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:48:14.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:48:14.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:48:14.041 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T16:48:14.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:48:14.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:48:14.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:48:14.354 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:48:14.351+0000 7f5b63531540 -1 osd.2 36 log_to_monitors true 2024-09-18T16:48:15.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:48:15.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:48:15.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:48:15.355 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T16:48:15.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:48:15.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:48:15.722 INFO:tasks.workunit.client.0.smithi046.stderr:osd.2 up in weight 1 up_from 41 up_thru 0 down_at 37 last_clean_interval [34,36) [v2:127.0.0.1:6818/682177,v1:127.0.0.1:6819/682177] [v2:127.0.0.1:6820/682177,v1:127.0.0.1:6821/682177] exists,up 188f8efd-f9a6-40e9-8807-17d49af0200d 2024-09-18T16:48:15.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:48:15.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:48:15.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:48:15.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T16:48:15.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:48:15.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:48:15.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:48:15.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:48:15.723 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:48:15.724 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:48:15.724 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:48:15.724 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:48:15.724 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:48:15.724 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:48:15.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:48:15.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:48:15.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:48:15.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:48:15.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:48:16.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:48:16.244 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:48:16.244 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:48:16.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:48:16.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:48:16.244 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:48:16.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836512 2024-09-18T16:48:16.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836512 2024-09-18T16:48:16.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836512' 2024-09-18T16:48:16.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:48:16.427 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:48:16.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051593 2024-09-18T16:48:16.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051593 2024-09-18T16:48:16.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836512 1-124554051593' 2024-09-18T16:48:16.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:48:16.558 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:48:16.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=176093659138 2024-09-18T16:48:16.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 176093659138 2024-09-18T16:48:16.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836512 1-124554051593 2-176093659138' 2024-09-18T16:48:16.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:48:16.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836512 2024-09-18T16:48:16.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:48:16.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:48:16.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836512 2024-09-18T16:48:16.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:48:16.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836512 2024-09-18T16:48:16.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-09-18T16:48:16.695 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836512 2024-09-18T16:48:16.696 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:48:16.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836511 -lt 21474836512 2024-09-18T16:48:16.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:48:17.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:48:17.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:48:18.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836513 -lt 21474836512 2024-09-18T16:48:18.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:48:18.344 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-124554051593 2024-09-18T16:48:18.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:48:18.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:48:18.347 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:48:18.347 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-124554051593 2024-09-18T16:48:18.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051593 2024-09-18T16:48:18.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 124554051593' 2024-09-18T16:48:18.349 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 124554051593 2024-09-18T16:48:18.349 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:48:18.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051594 -lt 124554051593 2024-09-18T16:48:18.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:48:18.689 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-176093659138 2024-09-18T16:48:18.689 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:48:18.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:48:18.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-176093659138 2024-09-18T16:48:18.692 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:48:18.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=176093659138 2024-09-18T16:48:18.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 176093659138' 2024-09-18T16:48:18.694 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 176093659138 2024-09-18T16:48:18.694 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:48:19.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 176093659138 -lt 176093659138 2024-09-18T16:48:19.023 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:48:19.024 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:48:19.024 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:48:19.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:48:19.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:48:19.432 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:48:19.432 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:48:19.432 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:48:19.432 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:48:19.433 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:48:19.433 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:48:19.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:48:19.774 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:48:19.774 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:48:19.775 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:48:20.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:48:20.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:48:20.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:48:20.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:68: TEST_recover_unexpected: objectstore_tool td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-attr _ td/osd-unexpected-clone/_ 2024-09-18T16:48:20.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-unexpected-clone 2024-09-18T16:48:20.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T16:48:20.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-18T16:48:20.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T16:48:20.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-attr _ td/osd-unexpected-clone/_ 2024-09-18T16:48:20.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-unexpected-clone 2024-09-18T16:48:20.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T16:48:20.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-18T16:48:20.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T16:48:20.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-unexpected-clone TERM osd.2 2024-09-18T16:48:20.137 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:48:20.137 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:48:20.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:48:20.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:48:20.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:48:20.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:48:20.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-attr _ td/osd-unexpected-clone/_ 2024-09-18T16:48:20.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-unexpected-clone 2024-09-18T16:48:20.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T16:48:20.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-18T16:48:20.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T16:48:20.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-unexpected-clone/2 2024-09-18T16:48:20.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-unexpected-clone/2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-attr _ td/osd-unexpected-clone/_ 2024-09-18T16:48:22.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-unexpected-clone 2 2024-09-18T16:48:22.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-unexpected-clone 2024-09-18T16:48:22.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T16:48:22.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T16:48:22.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T16:48:22.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-unexpected-clone/2 2024-09-18T16:48:22.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3fdc75b7-dab6-4baf-bc58-f78ba81ca67a --auth-supported=none --mon-host=127.0.0.1:7144 ' 2024-09-18T16:48:22.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:48:22.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:48:22.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:48:22.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/2' 2024-09-18T16:48:22.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/2/journal' 2024-09-18T16:48:22.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T16:48:22.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T16:48:22.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2024-09-18T16:48:22.517 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T16:48:22.517 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:48:22.517 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:48:22.518 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:48:22.518 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:48:22.518 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.671598 2024-09-18T16:48:22.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.671598/$cluster-$name.asok' 2024-09-18T16:48:22.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.671598/$cluster-$name.asok' 2024-09-18T16:48:22.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:48:22.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2024-09-18T16:48:22.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2024-09-18T16:48:22.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:48:22.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:48:22.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:48:22.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:48:22.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T16:48:22.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T16:48:22.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-unexpected-clone/2 2024-09-18T16:48:22.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T16:48:22.521 INFO:tasks.workunit.client.0.smithi046.stderr:start osd.2 2024-09-18T16:48:22.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=3fdc75b7-dab6-4baf-bc58-f78ba81ca67a --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/2 --osd-journal=td/osd-unexpected-clone/2/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.671598/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:48:22.521 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-unexpected-clone/2/whoami 2024-09-18T16:48:22.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T16:48:22.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T16:48:22.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T16:48:22.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T16:48:22.549 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:48:22.546+0000 7fd125b7d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:48:22.552 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:48:22.546+0000 7fd125b7d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:48:22.553 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:48:22.550+0000 7fd125b7d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:48:22.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T16:48:22.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:48:22.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T16:48:22.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:48:22.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:48:22.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:48:22.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:48:22.843 INFO:tasks.workunit.client.0.smithi046.stderr:0 2024-09-18T16:48:22.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:48:22.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:48:23.112 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:48:23.106+0000 7fd125b7d540 -1 Falling back to public interface 2024-09-18T16:48:23.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:48:24.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:48:24.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:48:24.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:48:24.203 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:48:24.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:48:24.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:48:24.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:48:25.543 INFO:tasks.workunit.client.0.smithi046.stderr:2024-09-18T16:48:25.538+0000 7fd125b7d540 -1 osd.2 42 log_to_monitors true 2024-09-18T16:48:25.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:48:25.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:48:25.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:48:25.568 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-09-18T16:48:25.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:48:25.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:48:25.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:48:26.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:48:26.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:48:26.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:48:26.973 INFO:tasks.workunit.client.0.smithi046.stderr:3 2024-09-18T16:48:26.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:48:26.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:48:27.293 INFO:tasks.workunit.client.0.smithi046.stderr:osd.2 up in weight 1 up_from 46 up_thru 0 down_at 43 last_clean_interval [41,42) [v2:127.0.0.1:6818/683531,v1:127.0.0.1:6819/683531] [v2:127.0.0.1:6820/683531,v1:127.0.0.1:6821/683531] exists,up 188f8efd-f9a6-40e9-8807-17d49af0200d 2024-09-18T16:48:27.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:48:27.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:48:27.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:48:27.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T16:48:27.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:48:27.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:48:27.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:48:27.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T16:48:27.294 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:48:27.295 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:48:27.295 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:48:27.296 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:48:27.296 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:48:27.296 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:48:27.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:48:27.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:48:27.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:48:27.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:48:27.504 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:48:27.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:48:27.830 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-09-18T16:48:27.830 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-09-18T16:48:27.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:48:27.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:48:27.830 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:48:27.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836516 2024-09-18T16:48:27.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836516 2024-09-18T16:48:27.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836516' 2024-09-18T16:48:27.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:48:27.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:48:28.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051597 2024-09-18T16:48:28.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051597 2024-09-18T16:48:28.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836516 1-124554051597' 2024-09-18T16:48:28.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:48:28.094 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:48:28.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=197568495618 2024-09-18T16:48:28.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 197568495618 2024-09-18T16:48:28.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836516 1-124554051597 2-197568495618' 2024-09-18T16:48:28.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:48:28.225 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836516 2024-09-18T16:48:28.225 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:48:28.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:48:28.227 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836516 2024-09-18T16:48:28.227 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:48:28.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836516 2024-09-18T16:48:28.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-09-18T16:48:28.229 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.0 seq 21474836516 2024-09-18T16:48:28.230 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:48:28.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836515 -lt 21474836516 2024-09-18T16:48:28.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:48:29.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:48:29.532 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:48:29.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836516 -lt 21474836516 2024-09-18T16:48:29.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:48:29.843 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-124554051597 2024-09-18T16:48:29.843 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:48:29.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:48:29.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-124554051597 2024-09-18T16:48:29.847 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:48:29.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051597 2024-09-18T16:48:29.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 124554051597' 2024-09-18T16:48:29.848 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.1 seq 124554051597 2024-09-18T16:48:29.849 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:48:30.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051597 -lt 124554051597 2024-09-18T16:48:30.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:48:30.174 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-197568495618 2024-09-18T16:48:30.174 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:48:30.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:48:30.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-197568495618 2024-09-18T16:48:30.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:48:30.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=197568495618 2024-09-18T16:48:30.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 197568495618' 2024-09-18T16:48:30.178 INFO:tasks.workunit.client.0.smithi046.stderr:waiting osd.2 seq 197568495618 2024-09-18T16:48:30.178 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:48:30.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 197568495618 -lt 197568495618 2024-09-18T16:48:30.532 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:48:30.532 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:48:30.533 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:48:30.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:48:30.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:48:30.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:48:30.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:48:30.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:48:30.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:48:30.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:48:30.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:48:31.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:48:31.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:48:31.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:48:31.279 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:48:31.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:48:31.636 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:48:31.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:48:31.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:70: TEST_recover_unexpected: sleep 5 2024-09-18T16:48:36.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:72: TEST_recover_unexpected: ceph pg repair 1.0 2024-09-18T16:48:36.936 INFO:tasks.workunit.client.0.smithi046.stderr:instructing pg 1.0 on osd.1 to repair 2024-09-18T16:48:36.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:74: TEST_recover_unexpected: sleep 10 2024-09-18T16:48:46.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:76: TEST_recover_unexpected: ceph log last 2024-09-18T16:48:47.311 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T16:48:38.145607+0000 osd.1 (osd.1) 2 : cluster [ERR] 1.0 shard 0 1:602f83fe:::foo:1 : missing 2024-09-18T16:48:47.311 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T16:48:38.145611+0000 osd.1 (osd.1) 3 : cluster [ERR] 1.0 shard 1 1:602f83fe:::foo:1 : missing 2024-09-18T16:48:47.312 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T16:48:38.145692+0000 osd.1 (osd.1) 4 : cluster [ERR] repair 1.0 1:602f83fe:::foo:1 : is an unexpected clone 2024-09-18T16:48:47.312 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T16:48:38.146084+0000 osd.1 (osd.1) 5 : cluster [ERR] 1.0 repair 1 missing, 0 inconsistent objects 2024-09-18T16:48:47.312 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T16:48:38.146151+0000 osd.1 (osd.1) 6 : cluster [ERR] 1.0 repair 3 errors, 2 fixed 2024-09-18T16:48:47.312 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T16:48:38.346873+0000 osd.1 (osd.1) 8 : cluster [ERR] 1.0 shard 1 soid 1:602f83fe:::foo:1 : candidate size 0 info size 13116 mismatch 2024-09-18T16:48:47.312 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T16:48:38.346877+0000 osd.1 (osd.1) 9 : cluster [ERR] 1.0 shard 0 soid 1:602f83fe:::foo:1 : data_digest 0x8f0babad != data_digest 0x7a63aba4 from auth oi 1:602f83fe:::foo:1(20'2 client.4252.0:1 dirty|data_digest s 13116 uv 1 dd 7a63aba4 alloc_hint [0 0 0]) 2024-09-18T16:48:47.312 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T16:48:38.346880+0000 osd.1 (osd.1) 10 : cluster [ERR] 1.0 shard 1 soid 1:602f83fe:::foo:1 : data_digest 0xffffffff != data_digest 0x8f0babad from shard 0, data_digest 0xffffffff != data_digest 0x7a63aba4 from auth oi 1:602f83fe:::foo:1(20'2 client.4252.0:1 dirty|data_digest s 13116 uv 1 dd 7a63aba4 alloc_hint [0 0 0]), size 0 != size 13116 from auth oi 1:602f83fe:::foo:1(20'2 client.4252.0:1 dirty|data_digest s 13116 uv 1 dd 7a63aba4 alloc_hint [0 0 0]), size 0 != size 13116 from shard 0 2024-09-18T16:48:47.312 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T16:48:38.346885+0000 osd.1 (osd.1) 11 : cluster [ERR] 1.0 soid 1:602f83fe:::foo:1 : data_digest 0x7a63aba4 != data_digest 0x8f0babad from shard 0 2024-09-18T16:48:47.312 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T16:48:38.346973+0000 osd.1 (osd.1) 12 : cluster [ERR] deep-scrub 1.0 1:602f83fe:::foo:1 : is an unexpected clone 2024-09-18T16:48:47.312 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T16:48:38.347445+0000 osd.1 (osd.1) 13 : cluster [ERR] 1.0 deep-scrub 0 missing, 1 inconsistent objects 2024-09-18T16:48:47.312 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T16:48:38.347451+0000 osd.1 (osd.1) 14 : cluster [ERR] 1.0 deep-scrub 4 errors 2024-09-18T16:48:47.312 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T16:48:43.594436+0000 mon.a (mon.0) 361 : cluster [ERR] Health check failed: 4 scrub errors (OSD_SCRUB_ERRORS) 2024-09-18T16:48:47.312 INFO:tasks.workunit.client.0.smithi046.stdout:2024-09-18T16:48:43.594532+0000 mon.a (mon.0) 362 : cluster [ERR] Health check failed: Possible data damage: 1 pg inconsistent (PG_DAMAGED) 2024-09-18T16:48:47.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:79: TEST_recover_unexpected: timeout 60 ceph tell osd.0 version 2024-09-18T16:48:47.483 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:48:47.484 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "17.2.7-1645-g7061ea6c", 2024-09-18T16:48:47.484 INFO:tasks.workunit.client.0.smithi046.stdout: "release": "quincy", 2024-09-18T16:48:47.484 INFO:tasks.workunit.client.0.smithi046.stdout: "release_type": "stable" 2024-09-18T16:48:47.484 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:48:47.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:80: TEST_recover_unexpected: timeout 60 ceph tell osd.1 version 2024-09-18T16:48:47.619 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:48:47.619 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "17.2.7-1645-g7061ea6c", 2024-09-18T16:48:47.619 INFO:tasks.workunit.client.0.smithi046.stdout: "release": "quincy", 2024-09-18T16:48:47.619 INFO:tasks.workunit.client.0.smithi046.stdout: "release_type": "stable" 2024-09-18T16:48:47.619 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:48:47.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:81: TEST_recover_unexpected: timeout 60 ceph tell osd.2 version 2024-09-18T16:48:47.751 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-09-18T16:48:47.752 INFO:tasks.workunit.client.0.smithi046.stdout: "version": "17.2.7-1645-g7061ea6c", 2024-09-18T16:48:47.752 INFO:tasks.workunit.client.0.smithi046.stdout: "release": "quincy", 2024-09-18T16:48:47.752 INFO:tasks.workunit.client.0.smithi046.stdout: "release_type": "stable" 2024-09-18T16:48:47.752 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-09-18T16:48:47.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:35: run: teardown td/osd-unexpected-clone 2024-09-18T16:48:47.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-unexpected-clone 2024-09-18T16:48:47.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:48:47.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-unexpected-clone KILL 2024-09-18T16:48:47.763 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:48:47.763 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:48:47.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:48:47.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:48:47.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:48:47.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:48:47.878 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:48:47.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:48:47.881 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:48:47.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:48:47.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:48:47.883 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:48:47.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:48:47.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:48:47.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:48:47.886 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:48:47.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:48:47.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:48:47.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-unexpected-clone 2024-09-18T16:48:47.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:48:47.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:48:47.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.671598 2024-09-18T16:48:47.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.671598 2024-09-18T16:48:47.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:48:47.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:48:47.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T16:48:47.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-unexpected-clone 0 2024-09-18T16:48:47.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-unexpected-clone 2024-09-18T16:48:47.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T16:48:47.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-unexpected-clone KILL 2024-09-18T16:48:47.914 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:48:47.914 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:48:47.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:48:47.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:48:47.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:48:47.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:48:47.919 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:48:47.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:48:47.921 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:48:47.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:48:47.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:48:47.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:48:47.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:48:47.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:48:47.926 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:48:47.926 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:48:47.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:48:47.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T16:48:47.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-unexpected-clone 2024-09-18T16:48:47.934 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:48:47.934 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:48:47.934 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.671598 2024-09-18T16:48:47.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.671598 2024-09-18T16:48:47.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:48:47.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:48:47.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T16:48:47.935 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T16:48:47.935 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T16:48:47.951 INFO:tasks.workunit:Stopping ['scrub'] on client.0... 2024-09-18T16:48:47.951 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-18T16:48:48.257 DEBUG:teuthology.parallel:result is None 2024-09-18T16:48:48.258 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-18T16:48:48.272 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-18T16:48:48.272 DEBUG:teuthology.orchestra.run.smithi046:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-18T16:48:48.329 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-18T16:48:48.329 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-18T16:48:48.342 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-18T16:48:48.342 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-18T16:48:48.383 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-18T16:48:48.391 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-18T16:48:48.392 DEBUG:teuthology.orchestra.run.smithi046:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T16:48:48.425 INFO:teuthology.orchestra.run.smithi046.stdout: remote refid st t when poll reach delay offset jitter 2024-09-18T16:48:48.425 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================== 2024-09-18T16:48:48.425 INFO:teuthology.orchestra.run.smithi046.stdout:*hv01.front.sepi 67.205.162.81 3 u 108 128 377 0.212 0.531 0.107 2024-09-18T16:48:48.425 INFO:teuthology.orchestra.run.smithi046.stdout:+hv02.front.sepi 63.231.80.2 3 u 56 128 377 0.165 1.270 0.191 2024-09-18T16:48:48.425 INFO:teuthology.orchestra.run.smithi046.stdout:+hv03.front.sepi 74.6.168.72 3 u 37 128 377 0.186 -0.048 0.208 2024-09-18T16:48:48.425 INFO:teuthology.orchestra.run.smithi046.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2024-09-18T16:48:48.426 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-18T16:48:48.436 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-18T16:48:48.436 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-18T16:48:48.465 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-18T16:48:48.497 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-18T16:48:48.531 INFO:teuthology.task.internal:Duration was 9383.779857 seconds 2024-09-18T16:48:48.531 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-18T16:48:48.564 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-18T16:48:48.564 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-18T16:48:48.617 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-18T16:48:48.617 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi046.front.sepia.ceph.com 2024-09-18T16:48:48.617 DEBUG:teuthology.orchestra.run.smithi046:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-18T16:48:48.629 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-18T16:48:48.629 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T16:48:48.717 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-18T16:48:48.717 DEBUG:teuthology.orchestra.run.smithi046:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-18T16:48:48.783 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-18T16:48:48.791 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-18T16:48:48.791 DEBUG:teuthology.orchestra.run.smithi046:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-18T16:48:48.808 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-18T16:48:48.817 DEBUG:teuthology.orchestra.run.smithi046:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:48:48.828 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = core 2024-09-18T16:48:48.844 DEBUG:teuthology.orchestra.run.smithi046:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:48:48.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T16:48:48.890 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-18T16:48:48.899 INFO:teuthology.task.internal:Transferring archived files... 2024-09-18T16:48:48.901 DEBUG:teuthology.misc:Transferring archived files from smithi046:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911098/remote/smithi046 2024-09-18T16:48:48.902 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-18T16:48:48.935 INFO:teuthology.task.internal:Removing archive directory... 2024-09-18T16:48:48.935 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-18T16:48:48.942 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-18T16:48:48.965 INFO:teuthology.task.internal:Not uploading archives. 2024-09-18T16:48:48.966 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-18T16:48:48.998 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-18T16:48:48.999 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-18T16:48:49.006 INFO:teuthology.orchestra.run.smithi046.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 16:48 /home/ubuntu/cephtest 2024-09-18T16:48:49.009 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-18T16:48:49.032 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-18T16:48:49.102 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} duration: 9383.77985715866 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-18T16:48:49.102 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T16:48:49.185 INFO:teuthology.run:pass